I'm seeing the same exact issue as OP as far as I can tell, in Xcode 16 beta 6 - I submitted feedback FB15045309

Re-posted at https://developer.apple.com/forums/thread/742568 due to accidental resolution

I know this is a year old, but I was just playing around with something similar in Xcode Cloud and spent a mildly embarrassing amount of time before finding https://developer.apple.com/videos/play/wwdc2021/10269. Taking a look at around 11:50-12:30 in that video, and putting some pieces together, I was able to get my baseline files to exist by downloading them from an S3 bucket during the pre-xcodebuild stage. Specifically, for test jobs, it seems that there are two actions - build-for-testing and test-without-building. Since test-without-building happens on different machines than build-for-testing, I specifically had to sync down the baseline images from S3 like this in pre-xcodebuild: if [ $CI_XCODEBUILD_ACTION = 'test-without-building' ] then # Procure your baseline images here fi That way, the baseline images were actually recognized as being present during the actual test run when the new images were being generated. Basically, my takeaway is that if you want any of your images of any sort to exist when the actual tests are being run, you're gonna have to get them in the pre-xcodebuild stage for that test-without-building action.