3 Replies
      Latest reply on Aug 2, 2019 1:09 AM by eskimo
      jake_prickett8212 Level 1 Level 1 (0 points)

        Hello everyone!

         

        I work for a company where we have a relatively large XCUI Test suite. When we utilize the Xcode simulator for running the suite, everything works extremely well for Software Engineers - everyone is happy!

         

        The problem comes when we attempt to run the suite using real devices from a Third Party Provider's cloud-based lab. The outcome is polar opposite of what we see on the simulator. For example - locally 100% pass, on the real devices ~80% pass.

         

        A few examples of the failures would be:

        • Application not able to find element randomly
        • Delays in Call prompts being displayed
          • Although it is properly handled programatically and works on local device
        • Application randomly becoming unresponsive

         

        I wanted to open this up to get help on two things:

        1. Technical differences/benefits of running on Simulator vs. Device (any insight into why they're so inconsistent)
        2. See if others have faced this issue and have advice

         

        Thanks in advance for your help!

        • Re: XCUI on Simulator vs. Real Device
          eskimo Apple Staff Apple Staff (12,325 points)

          I don’t have any direct experience with this but I do have one suggestion: You need to tease apart these cases:

          • Failures in your app code when running on real hardware

          • Failures related to UI tests running on real hardware

          • Failures introduced by your third-party testing provider

          As a first step, I recommend that you create a small test setup locally using Xcode bots.  If you see big differences in failure rate between your local test setup and your third-party testing provider, you should escalate this issue to them.  OTOH, if your local test setup has a similar failure rate, you know the third-party testing provider isn’t at fault.

          Share and Enjoy

          Quinn “The Eskimo!”
          Apple Developer Relations, Developer Technical Support, Core OS/Hardware
          let myEmail = "eskimo" + "1" + "@apple.com"

            • Re: XCUI on Simulator vs. Real Device
              jake_prickett8212 Level 1 Level 1 (0 points)

              That sounds like a solid approach, apprecieate the tip!

               

              As for the technical differences/benefits of running on Simulator vs. Device, more specifically what benefits do teams get by running their XCUI tests on real devices rather than simulators? We are being asked to utilize the third-party provider, so now it'll be much more difficult to maintain a successful, reliable test suite that works on both Simulators and Real devices.

               

              Would it be recommended to pick one or the other?

                • Re: XCUI on Simulator vs. Real Device
                  eskimo Apple Staff Apple Staff (12,325 points)

                  Ultimately you need to test on real devices.  The simulator is good, but it’s not perfect.  The most obvious difference is in performance (although these days, the performance advantage may well be on the real device side) but there are others.  Personally, I support a bunch of technologies, like networking and multitasking, where the simulator is just not up to the job.

                  However, on the specific topic of UI testing, I don’t have a lot of practical advice.  I don’t do much UI, and thus I don’t do much UI testing.

                  Share and Enjoy

                  Quinn “The Eskimo!”
                  Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                  let myEmail = "eskimo" + "1" + "@apple.com"