Struggling every single day to get consistent testing results, and over the past few days I've run into a whole series of brand new exciting issues!
Why did these new issues show up on a branch of code that never changes? Who knows! They seemed to work OK when I switched to Xcode 9.2... until they didn't, for mysterious reasons. I don't get it.
I have two tests, triggered hourly by an XCBot, running on 1 10.3 and 3 11.2 simulators. I've tried running tests in parallel and not.
I can run these tests 20 times on my XCBot and get 20 different results.
Run each set of tests manually in Xcode on the same codebase (command-U in Xcode) and every test passes perfectly on every simulator.
The list of weirdo problems:
- Test Assertion: Failed to terminate com.****.****:3850 after 60.0s; state is still `Running Foreground` (this is the big one for me; one I've never seen before)
- Test Assertion Launch session expired
- Explicit assertions made by me are reported with the cryptic "Test Assertion Crash: UITests-Runner: implicit closure #1 in UITests" rather than my assertion statement (although sometimes I get both and sometimes I only get my own message)
- An assertion that the count of a particular on-screen element should be greater than 0 fails even though the automated screenshot clearly shows the elements being counted and those same tests run perfectly outside of the XCBot environment.
I'm at a total loss here.
