Xcode 15.3+ bug: framework tests executed in fresh simulator fail to write data to disk

I have filed this as FB13722352. I am sharing it here because I haven't seen it mentioned anywhere online yet and am curious if anyone else has run into it.

In Xcode 15.3+, writing data to disk fails when running the tests for a Framework project in a fresh simulator. Specifically, if the selected simulator has never been launched before (i.e. is newly-created), any test execution that attempts to write data into the NSDocumentDirectory directory will fail for a period of time after the simulator is first launched (I've observed between 10s and 20s). After that period of time, the same data write action will succeed. It appears that Xcode 15.3+ is starting test execution too soon, without waiting a sufficient amount of time for the Simulator to fully boot. This issue does not occur in Xcode 15.2 or prior versions.

Since the issue only appears in a fresh (never-before booted) simulator, it is likely to pop up consistently in CI test runs (where simulators are not re-used). This can cause confusion because the same test would not fail locally when re-using an existing simulator.

When the issue appears, the file write API returns the following error:

Domain=NSCocoaErrorDomain Code=4 "The folder “testFile” doesn’t exist." UserInfo={NSFilePath=[...]/data/Documents/testFile, NSUserStringVariant=Folder, NSUnderlyingError= {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"} }

Reproduction Steps:

1. Open Xcode 15.3 or 15.4. Make sure Simulator.app is closed.
2. Using the "Devices and Simulators" window, create a new iPhone 15 Pro simulator with iOS 17.4 (other devices and OS versions work as well). Do not launch this new simulator.
3. Create a new Framework project and add a test that performs and then checks the output of a data write to the Document directory (see example test code below).
4. Select the new simulator (created in step 2) as the test run target and run the test.

Here's an example test that fails in the scenario outlined above:

- (void)testBasicRepro {
    NSString *testString = @"Hello, World!";
    NSData *data = [testString dataUsingEncoding:NSUnicodeStringEncoding];

    // Get documents directory
    NSURL *url = [[[NSFileManager defaultManager] URLsForDirectory:NSDocumentDirectory inDomains:NSUserDomainMask] lastObject];
    NSURL *testFileURL = [url URLByAppendingPathComponent:@"testFile"];

    // Write the data
    NSError *error;
    bool result = [data writeToURL:testFileURL options:NSDataWritingAtomic error:&error];

    // Check if it was successful
    XCTAssertTrue(result);
    XCTAssertNil(error);
    XCTAssertTrue([[NSFileManager defaultManager] fileExistsAtPath:testFileURL.path]);
}

Workaround

The workaround that I have come up with is to create a test that runs first (by disabling parallelization and randomization, and making sure the test class filename is alphabetically first). Alternatively, it could be called from the setUp method in any test files that are affected. This test performs a data write and checks the result in a loop in order to block until the data write succeeds (i.e. the Simulator is sufficiently booted for data write operations to complete).

- (void)testWorkaroundBug {
    NSString *testString = @"Hello, World!";
    NSData *data = [testString dataUsingEncoding:NSUnicodeStringEncoding];
    NSError *error;

    // Get documents directory
    NSURL *documentsURL = [[[NSFileManager defaultManager] URLsForDirectory:NSDocumentDirectory inDomains:NSUserDomainMask] lastObject];
    NSURL *testFileURL;

    NSDate *startTime = [NSDate date];
    NSLog(@"Starting test at %@", startTime);
    for (int i = 0; i < 120; i++) {
        // Create unique URL
        testFileURL = [documentsURL URLByAppendingPathComponent:[NSString stringWithFormat:@"testFile-%@", @(i)]];
        // Write the data
        BOOL success = [data writeToURL:testFileURL options:NSDataWritingAtomic error:&error];

        // Check if it exists
        if (success && [[NSFileManager defaultManager] fileExistsAtPath:testFileURL.path]) {
            NSLog(@"Test file %@ was created successfully! Elapsed time %@s", @(i), @(fabs([startTime timeIntervalSinceNow])));
            return;
        }
        else {
            NSLog(@"Test file %@ was not created. Error: %@. Sleeping for 0.5s and trying again.", @(i), error);
            [NSThread sleepForTimeInterval:0.5];
        }
    }
}