XCUI on Simulator vs. Real Device

Hello everyone!


I work for a company where we have a relatively large XCUI Test suite. When we utilize the Xcode simulator for running the suite, everything works extremely well for Software Engineers - everyone is happy!


The problem comes when we attempt to run the suite using real devices from a Third Party Provider's cloud-based lab. The outcome is polar opposite of what we see on the simulator. For example - locally 100% pass, on the real devices ~80% pass.


A few examples of the failures would be:

  • Application not able to find element randomly
  • Delays in Call prompts being displayed

    Although it is properly handled programatically and works on local device

  • Application randomly becoming unresponsive


I wanted to open this up to get help on two things:

  1. Technical differences/benefits of running on Simulator vs. Device (any insight into why they're so inconsistent)
  2. See if others have faced this issue and have advice 🙂


Thanks in advance for your help!

Up vote post of jake_prickett8212 Down vote post of jake_prickett8212
1.9k views
Posted by
jake_prickett8212
Reply
Add a Comment

Replies

I don’t have any direct experience with this but I do have one suggestion: You need to tease apart these cases:

  • Failures in your app code when running on real hardware

  • Failures related to UI tests running on real hardware

  • Failures introduced by your third-party testing provider

As a first step, I recommend that you create a small test setup locally using Xcode bots. If you see big differences in failure rate between your local test setup and your third-party testing provider, you should escalate this issue to them. OTOH, if your local test setup has a similar failure rate, you know the third-party testing provider isn’t at fault.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware 

let myEmail = "eskimo" + "1" + "@apple.com"
Posted by
eskimo
Up vote reply of eskimo Down vote reply of eskimo
Add a Comment

That sounds like a solid approach, apprecieate the tip!


As for the technical differences/benefits of running on Simulator vs. Device, more specifically what benefits do teams get by running their XCUI tests on real devices rather than simulators? We are being asked to utilize the third-party provider, so now it'll be much more difficult to maintain a successful, reliable test suite that works on both Simulators and Real devices.


Would it be recommended to pick one or the other?

Posted by
jake_prickett8212
Up vote reply of jake_prickett8212 Down vote reply of jake_prickett8212
Add a Comment

Ultimately you need to test on real devices. The simulator is good, but it’s not perfect. The most obvious difference is in performance (although these days, the performance advantage may well be on the real device side) but there are others. Personally, I support a bunch of technologies, like networking and multitasking, where the simulator is just not up to the job.

However, on the specific topic of UI testing, I don’t have a lot of practical advice. I don’t do much UI, and thus I don’t do much UI testing.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware 

let myEmail = "eskimo" + "1" + "@apple.com"
Posted by
eskimo
Up vote reply of eskimo Down vote reply of eskimo
Add a Comment