Xcode 7.2 bug: every other test case being skipped

Has anyone else seen this? I'm consistently having XCTest hang during every other test run on a test device (haven't tried simulator), with the test case failing and resulting in the following error:


UI Testing Failure - App state for <XCUIApplicationProcess: 0x13ded1930 (null) (1267)> is XCApplicationStateRunningActive (3), still not XCApplicationStateNotRunning (1)


Anyone else seeing this? It appears to be happening with:

http://stackoverflow.com/questions/34325297/xcode-ui-tests-are-failing-because-of-xcapplicationstaterunningactive-on-physica

Replies

Yes I do see this. I'm running on simulator.


Apple engineers:

Can I really trust that you have an extensive test suite where you really try to stress test your XCTest/XCUITest frameworks? If you don't have that, please add such suites immediately. To me the test frameworks seem to be instable. They might work most of the times for 5 TCs running for one second in a suite, but is it really 100% stable when you add 100 TCs in a suite? Can I trust that each TC will return the same output every time without errors like the one above? Can I trust that I can add a 10 minute graphical TC in XCUITest that terminates with the same output every time without test framework problems? Can I add 50 10 minute graphical TCs that runs over night? Etc.


I try to implement such test suites for my app, to guarantee the quality of my software. I have hard times to do that. I'm currently running 1 (one) TC for ~1 minute, and that TC is passing 80% of the runs. Now it terminated with the error above. Don't know what will be the reason next time...


Please, please do extensive tests of the test frameworks on both devices and simulators.

Yep, I've seen it too - it's quite frustrating. Is it happening on a device too, or just on the simulator?


Something I've noticed: whatever tests are run, whatever order they are in, if something does fail it always seems to be the second test. Weird.

We're getting the exact same error on our continuous integration server on `XCUIApplication().terminate()`:


UI Testing Failure - App state for <XCUIApplicationProcess: 0x7faefb904ff0 (null) (8356)> is XCApplicationStateRunningActive (3), still not XCApplicationStateNotRunning (1)


We're able to reproduce these errors locally by slowing our machines down.


These errors are seriously disrupting our workflow as they cause our continuous integration tests to never pass.

"We're able to reproduce these errors locally by slowing our machines down."

Haha, that explains why my Macbook Air is having these problems...

Seeing the same with my tests 😟

I would suggest that one uninstall 7.2 and reinstall Xcode from a 7.1.1 that installer, but my test devices have all been updated to 9.2, which Xcode 7.1.1 can't run tests on.

Not sure if it's every other test, but we're getting it too—everything is running fine locally, but on the CI server (either under CI or under Xcode running as a normal user):


Assertion Failure: UI Testing Failure - App state for <XCUIApplicationProcess: 0x155659460 (null) (814)> is XCApplicationStateRunningActive (3), still not XCApplicationStateNotRunning (1)


Submitting a radar.

I'm getting this as well. I've got ~60 UI tests and on my local dev machine it gets about halfway through the tests before this error starts happening. On the older CI box with OS X server on it it gets about 4 tests in before the error starts happening. Frustrating!

It's clear that the build machine is somehow resource constrained, but I can't work out the exact nature of the constraint. What I'd like to do is run some kind of process on my desktop machine which will eat the memory/cores/processing power/whatever and cause the build to fail locally as it does on CI, then I have a hope of debugging the **** thing.

I am seeing the same issue. I put in a dummy test between each of my real tests and every other test fails with that error. I named my tests with numbers so that the tests run in the same order each time. This happens on my local macbook pro

Same issue for me. I have only two tests, but the second one always fails. I'm running the UI Tests on device.

I'm seeing this issue also (Xcode 7.2).

I "fixed" this by giving the machine enough time to start the app, sleep(3) after every app.launch(). It's not elegant etc but it works reliably now.

Adding sleeps (even very long ones) does not resolve the problem for me. I see this problem every time I call terminate(). The test is otherwise executing correctly, the app does appear to terminate on the device, and if it wasn't for this assertion (which occurs in my tearDown() methods), my tests would succeed.

My radar has been closed as a dupe of 23920068.


Any ETA on an Xcode 7.2.1 or 7.3 fixing this?