Experiences integrating XCUITest with Jenkins?

Has anyone integrated Xcode 7's UI Testing features with their Jenkins CI system? What have your experiences been?


The setup itself is not too challenging, but the results haven't been great so far. Aside from the debilitating bug introduced in 7.2, a lot of UI Tests seem to spontaneously break intermittently from mysterious UI issues. That's less bad when running the tests directly from Xcode itself, where you can conveniently review the results afterwards thanks to the screenshots and XCTest logs. However, if you run the tests via xcodebuild as a Jenkins script, at most the results yield you a line containing the step where the tests fail (though I've also been piping the XCTest log into a logfile). It's definitely less helpful than Xcode, even though it seems like xcodebuild is still saving the screenshots somewhere, albeit in a disconnected manner.

Replies

XCUITest is incredibly flaky. With XCTest being open sourced, I can only hope XCUITest follows quickly. https://github.com/apple/swift-corelibs-xctest The current XCUITest is broken to the point where I question if anyone is having success using it in CI at scale.

I just wonder if anyone has tried to use XCUI as intended, i.e. as part of a CI implementation. How about with Xcode Bots, or with Bamboo, Travis CI, etc.?

I had a test suite running on Jenkins and it was just as flaky as it is when you run it on your machine. Not sure what the deal is though. Even when I reset the user settings between each test the app still stays in a state that is not as if a new user is using it. I think it will grow in the future and get better.

We have 115 UI Tests that runs daily on the jenkins server. Best result that we can get - ~5% fails. Every run around 10-20 tests failed for some misterious and random reasons, that can be reproduced on the local machine from time to time.

And a couple of days ago we even realized, that sometimes test failed but reports as succeded. So we had like a lot of 2-3 minutes tests, that were magically "passed" in 15 seconds.

As a result - UI Tests are quite unstable but still it is a very good instrument and I hope most of the issues will be fixed in the nearest future

I have tried XCUITest with Bots and my experience isn't any better than what you're reporting with Jenkins. Random failures when the tests are reliable running locally, different behaviour on simulator vs on device under CI, the 7.2 "every second test fails" bug you linked.

Am so glad I read this message. Setup Jenkins with Xcode plugin but getting fairly Flakey results.


So in terms of automation suites do people feel that UI Testing is less reliable than Appium and Calabash?