Xcode Cloud snapshot testing using ci_scripts directory

Question

Fogmeister OP

Created Oct ’22

Replies 2

Boosts 1

Views 2.5k

Participants 3

Setup

I've been experimenting with migrating to Xcode Cloud for our CI but I'm struggling with getting Snapshot Testing to work.

I've been involved in some discussions here... https://github.com/pointfreeco/swift-snapshot-testing/discussions/553

And I saw this site also which explains a little... https://wojciechkulik.pl/xcode/xcode-cloud-overview-and-setup

But so far I've been unsuccessful in getting it to work at all.

I'm using the swift-snapshot-testing library from PointFree https://github.com/pointfreeco/swift-snapshot-testing and I have overridden the assertSnapshot function to change the snapshot testing directory.

The function looks like this...

func snapshotDirectory(
	for file: StaticString,
	ciScriptsPathComponent: String = "ci_scripts",
	relativePathComponent: String = "Tests"
) -> String {
	var sourcePathComponents = URL(fileURLWithPath: "\(file)").pathComponents

	if let indexFolder = sourcePathComponents.firstIndex(of: relativePathComponent) {
		sourcePathComponents.insert("resources", at: indexFolder)
		sourcePathComponents.insert(ciScriptsPathComponent, at: indexFolder)
	}
	var pathsComponents: [String] = sourcePathComponents.dropLast()

	let fileUrl = URL(fileURLWithPath: "\(file)", isDirectory: false)
	let folderName = fileUrl.deletingPathExtension().lastPathComponent

	pathsComponents.append("__Snapshots__")
	pathsComponents.append(folderName)

	let directory = String(pathsComponents.joined(separator: "/").dropFirst())
	return directory
}

public func assertSnapshot<Value, Format>(
	matching value: @autoclosure () throws -> Value,
	as snapshotting: Snapshotting<Value, Format>,
	named name: String? = nil,
	record recording: Bool = false,
	timeout: TimeInterval = 5,
	file: StaticString = #file,
	testName: String = #function,
	line: UInt = #line
) {
	let failure = verifySnapshot(
		matching: try value(),
		as: snapshotting,
		named: name,
		record: recording,
		snapshotDirectory: snapshotDirectory(for: file),
		timeout: timeout,
		file: file,
		testName: testName
	)
	guard let message = failure else { return }
	XCTFail("\(message) snap: \(snapshotDirectoryUrl) file: \(file) ", file: file, line: line)
}

Essentially this takes my test file path... repoRoot/Tests/FeatureTests/FeatureTestFile.swift.

And injects some path component into it so that you end up with a directory path... repoRoot/ci_scripts/resources/Tests/FeatureTests/__Snapshots__/FeatureTestFile/.

And then the snapshot file will be located in that directory using the name of the test function with a suffix of .1.txt or .2.txt (etc... for each subsequent snapshot in each function). i.e. testSnapshotStuff.1.txt, testSnapshotStuff.2.txt.

Problem

This all works locally. And all the files are checked into GitHub.

But, when I run this on Xcode Cloud it fails the tests and tells me the files are not there.

Having added some logging in it is writing new snapshot files to where I am expecting them to be so it just looks like those files are not available to the Test environment.

This is where I read about putting them into the ci_scripts file at the root of the repo. Which is what I've done. Files in this directory are supposed to be copied into the test environments so that they can be accessed... but it seems that they're not being.

I have tried using ci_scripts/resources and ci_scripts/Artifacts but it's always the same. The files aren't there and the Xcode Cloud tests write those files there over time.

I'm running out of options of what to do with this now. I just want a way that I can access these snapshot files in the test environment on Xcode Cloud.

Any help would be much appreciated.

Thanks

Boost

Answer 1

alfonso_patio23 OP

Oct ’22

Did you create a system link so the screenshots get included by Xcode Cloud?

0

Answer 2

bfreib OP

Oct ’23

I know this is a year old, but I was just playing around with something similar in Xcode Cloud and spent a mildly embarrassing amount of time before finding https://developer.apple.com/videos/play/wwdc2021/10269. Taking a look at around 11:50-12:30 in that video, and putting some pieces together, I was able to get my baseline files to exist by downloading them from an S3 bucket during the pre-xcodebuild stage.

Specifically, for test jobs, it seems that there are two actions - build-for-testing and test-without-building. Since test-without-building happens on different machines than build-for-testing, I specifically had to sync down the baseline images from S3 like this in pre-xcodebuild:

if [ $CI_XCODEBUILD_ACTION = 'test-without-building' ]
then
    # Procure your baseline images here
fi

That way, the baseline images were actually recognized as being present during the actual test run when the new images were being generated. Basically, my takeaway is that if you want any of your images of any sort to exist when the actual tests are being run, you're gonna have to get them in the pre-xcodebuild stage for that test-without-building action.

0