"Randomly" failing tests

We have a test suite that passes when tests are run manually via Xcode.


But when run through Xcode Server on about 50 different simulators and devices (most combinations of iOS versions vs device type) we get "random" errors and failures. The devices/OS versions that it happens on always changes from intergration to intergration, but the tests that fail or error are typically the same few over and over again. This would suggest timing issues or similar, but it doesn't seem to be the case.


For example, some of them (#3 and #4) are cases of "1" is not not equal "1"-type failures.


Has anyone else had problems with sporadic Xcode Server CI test failures?


These two tests error (they never complete):


#1


Issue: Run test suite ResetPasswordHelperTests encountered an error (Test session exited(-1) without completing. If you believe this error repre.
Integration Number: 18.
Integration URL: https://[redacted].local/xcode/bots/D5DE641/integrations
Description: Run test suite ResetPasswordHelperTests encountered an error (Test session exited(-1) without completing. If you believe this error represents a bug, please attach the log file at /Library/Developer/XcodeServer/Integrations/Integration-a65a88452e866909928d000cda035549/Session-2015-07-21_14:41:17-SzQyLx.log).
Committers: Introduced 1 integration ago.


#2


Issue: test_send_request_reset_response_failed encountered an error (Test session exited(-1) without completing. If you believe this error represen.
Integration Number: 18.
Integration URL: https://[redacted].local/xcode/bots/D5DE641/integrations
Description: test_send_request_reset_response_failed encountered an error (Test session exited(-1) without completing. If you believe this error represents a bug, please attach the log file at /Library/Developer/XcodeServer/Integrations/Integration-a65a88452e866909928d000cda035549/Session-2015-07-21_14:40:33-Xtlc7c.log).
Committers: Introduced 3 integrations ago.



These tests fail:


#3


Issue: ((self.logEvents) equal to (@"/startLoading[https://www.mock.unitest.com/api/v1/search/?q=allBooksWithThisTerm&filter=book&sort=created_time.
File: SearchManagerTest.m.
Test Case: -[SearchManagerTest testNilCompletionBlock].
Integration Number: 17.
Integration URL: https://[redacted].local.local/xcode/bots/D5DE641/integrations
Description: ((self.logEvents) equal to (@"/startLoading[https://www.mock.unitest.com/api/v1/search/?q=allBooksWithThisTerm&filter=book&sort=created_time]/stopLoading[https://www.mock.unitest.com/api/v1/search/?q=allBooksWithThisTerm&filter=book&sort=created_time]")) failed: ("/startLoading[https://www.mock.unitest.com/api/v1/search/?q=allBooksWithThisTerm&filter=book&sort=created_time]/stopLoading[https://www.mock.unitest.com/api/v1/search/?q=allBooksWithThisTerm&filter=book&sort=created_time]") is not equal to ("/startLoading[https://www.mock.unitest.com/api/v1/search/?q=allBooksWithThisTerm&filter=book&sort=created_time]/stopLoading[https://www.mock.unitest.com/api/v1/search/?q=allBooksWithThisTerm&filter=book&sort=created_time]").



#4


Issue: ((self.logEvents) equal to (@"/taskStateDidChange[0]/taskStateDidChange[2]/taskStateDidChange[3]")) failed: ("/taskStateDidChange[0]/taskSta.
File: URLRequestTest.m.
Test Case: -[URLRequestTest testCancelAllTasks].
Integration Number: 17.
Integration URL: https://[redacted].local.local/xcode/bots/D5DE641/integrations
Description: ((self.logEvents) equal to (@"/taskStateDidChange[0]/taskStateDidChange[2]/taskStateDidChange[3]")) failed: ("/taskStateDidChange[0]/taskStateDidChange[2]/taskStateDidChange[3]") is not equal to ("/taskStateDidChange[0]/taskStateDidChange[2]/taskStateDidChange[3]").



#5


Issue: failed: caught "NSInvalidArgumentException", "+entityForName: nil is not a legal NSManagedObjectContext parameter searching for entity name .
File: CoreDataItemProviderTests.m.
Test Case: -[CoreDataItemProviderTests test_CoreDataItemProvider_should_not_crash].
Integration Number: 17.
Integration URL: https://[redacted].local.local/xcode/bots/D5DE641/integrations
Description: failed: caught "NSInvalidArgumentException", "+entityForName: nil is not a legal NSManagedObjectContext parameter searching for entity name '[Redacted]'"
Up vote post of WhyNotRealNames Down vote post of WhyNotRealNames
2.3k views
Posted by
WhyNotRealNames
Reply
Add a Comment

Replies

I've had lots of fun problems like that. They're usually caused by differences in the test environment, such as higher system load, slower CPU, etc. causing things to time out when they otherwise would have succeeded.


Network requests are by far the worst. I have some networking client code that does some rather neat tricks—restarting idempotent requests after a soft timeout (while continuing the original request), limiting the number of concurrent requests but allowing additional concurrent requests if the latency of pending requests gets too high, etc.—all of which result in more reliable network behavior, assuming the code works. However, that code needed tests, and writing those tests required creating a server that would take a parameter indicating how long it should wait before it responds, so that we could simulate arbitrarily slow connections with reproducible consistency.


My initial tests involved performing a long series of operations and checking to make sure various states changed at the right time. That approach worked beautifully until I tried it on an iOS device, at which point the timing differences between a native OS X app and an iOS app running on a device were so huge that the XCTest timing drifted enough to cause the tests to fail. And sporadic network delays between us and the Amazon test server seemed to be less than infrequent.

I ended up reworking the tests so that A. they're smaller in scope (fewer simultaneous waiting tasks) and B. they use a small daemon on localhost that responded to very simple HTTP requests and waited the requested amount of time. Ever since then, I haven't seen any random test failures.


The moral of the story is that networks ****, and any test that relies on a network not sucking is going to have a nonzero failure rate. Take the network out of the equation, either through mocking or through the use of a local webserver on the test machine, and your problems may just go away on their own. Either way, the problem is probably caused by timing differences. If it isn't, you'll need to add logging and figure out what's happening when those tests fail.

Posted by
dgatwood
Up vote reply of dgatwood Down vote reply of dgatwood
Add a Comment

I have the same kind of randomly failing tests, but none of my tests involve network operations. Some of them have expectations and other NSRunLoop stuff, so that might explain it, but some are just dead-simple "set a variable, check to see if the variable was set" tests. It's driving me crazy because:


1. My local machine and the Xcode server are the same mac mini, same configuration and same OS

2. Tests on my local machine always pass, but I almost always get 3-5 failing tests on the server machine, although they are almost always different tests.

3. These tests are only on an OSX target, no simulator.

Posted by
fedor
Up vote reply of fedor Down vote reply of fedor
Add a Comment