XCTests crash randomly when running on more than 2 simulators in parallel on iOS 13 simulators

When i try to run tests in parallel using

xcodebuild build-for-testing & xcodebuild test-without-building
via terminal (or in Jenkins) on more than 2 or 3 iOS 13.2 simulators, the tests start crashing randomly. The crashes start happening when the app does not get installed properly after running a few tests. Has anyone else been experiencing this issue and if there is a solution for this?

Replies

dpe.qa,

We've been experiencing very similar issues as well since moving up to Xcode 11 (both on Mojave and Catalina).

We are seeing deep DVTFramework and other CoreFoundation memory exceptions that are so far removed from user level code, I'm getting the funny feeling that the runtime environment is experiencing both memory and thread exhaustion. SInce we never *ever* saw any crashes this sever prior, I suspect the new iOS 13 APIs are expecting a less constrained environment. I have no formal proof for this assertion, but this pattern matches previous behavior I have seen on other *NIX based environments.


Now, with that being said, for S&Gs we temporarily moved our large amount of XCTest Classes to run serially and they all run just fine. Even on a poky little Mac Mini. Unfortunately an integration takes hours, not minutes. So this isn't a long term solution for us. We can run in parallel on our new spiffy Mac Book Pros just fine. However, even there we're seeing unexpected behavior of a Simulator Clone "failing" to start and thus gives us a false negative report at the end of the integration.


My suggestion is to dig into the run logs and see if you can find the actuall crash. Through the Bot system, I can navigate into the failed integration and actually download the entire .tgz of build/test logs. There should be 1+ .crash logs in that bundle that will immediatly give you the stack dump for the crash.

Yes, Tested with Xcode 11.3 and Xcode 11.4 beta. It is very much reproducible when building apps with a build-for-testing option.

Hi MCStoufer_veeva, dpe.qa, thirisangut

Have you been able to Symbolicate UITests-Runner crashes. Is there any way to get *UITests-Runner.app.dSYM file? In my case I can't see matching UUID with any of the available .dSYM(s) in the build-dir.

Any help will be much appreciated. Thank you.

Regards,
Soaurabh