Xcode 8.3.2 CI doesn't find any Simulator Devices

If I execute the following command ...

sudo xcrun xcscontrol --list-simulators

... the result is as follows ...

Found 0 total simulators


I guess that's why we cannot see any simulator in the bot configuration tab "Devices" if we choose "Test With: iOS - Specific iOS Devices", the dialog just shows "No devices found.". Most of the times anyways, sometimes if we leave the configuration dialog open for a very long time, a list of simulators suddenly appears - which is very strange.


However the "Devices" window of Xcode shows all Simulator Devices properly and so does the following command:

xcrun simctl list


Why is the Xcode CI system unable to find Simulator Devices on our system?

Is there any way to see a more verbose output of xcscontrol any why it seems to fail?


Don't know if this problem is related to the CouchDB problems of Xcode 8.3.2 described here: https://forums.developer.apple.com/thread/75581.


Xcode 8.3.2

macOS Server 5.3

Replies

> Most of the times anyways, sometimes if we leave the configuration dialog open for a very long time, a list of simulators suddenly appears - which is very strange.


Have to ask if you're giving background component downloads sufficient time to sign and install after moving to a new Xcode, meaning more than might seem normal, perhaps...could be the backend it taking it's time, penalizing your wait.


If a specific simulator device does not appear in the list of Selected Devices, I would confirm the sim installs on the server. This includes paths and permissions. Check the server logs for borked connections if you can time stamp the fallout.


File bugs against the tools if it seems out of your control, etc.


Good luck.

Having this exact same problem. My experience so far is logged here, which includes the "leave the configuration dialog open for a very long time, a list of simulators suddenly appears" issue as well:
https://forums.developer.apple.com/thread/76453


Just now I waited and finally saw a list of devices. Selected my devices, ran the integration and got a failure saying, "The devices configured for testing were not found." Edit Bot... > Devices and sure enough, "No devices found." again. Though the list was populated literally 30 seconds earlier.


Running Simulator on its own shows all devices, and I can launch/use them without trouble.
Inside Xcode > Product > Destination > iOS Simulators shows all expected simulators.
Inside Xcode > Window > Devices shows all expected simulators.


Only the bots can't see the simulators, and this is all-or-nothing.

  • When I see simulators, I see all simualtors.
  • When I get the error, I see zero simulators.

Quick update on my side:

Downgrading Xcode to 8.3.1 fixed the issue for me.

After some digging into Xcode Server 's node.js application, i believe i found a work-around solution. At least, it fixed everything for us: both the CPU usage problems with beam.cmp and node processes, and the trouble with devices and simulators being intermittently lost and unavailable for testing.


I wrote up a detailed description here (link broken deliberately to avoid this reply getting stuck in moderation queue):

h t t p s : // github.com/juce/xcs-tweaks


The quick summary is this:

It is an expensive operation for Xcode Server to retrieve information about available devices and simulators from CouchDB. So to speed things up, it caches that info in Redis. However, for whatever reason, reading that info back from Redis is even slower than querying CouchDB for it. It is during those reads from Redis that a node process starts using 100% cpu. This slow read causes timeouts, which in turn makes Xcode Server API unresponsive, which cascades to failing integrations, and causes problems when you try to edit a bot. So, the fix is to cache on a file on disk, instead of Redis.


For that you need to patch this file:

/Library/Developer/XcodeServer/CurrentXcodeSymlink/Contents/Developer/usr/share/xcs/xcsd/classes/deviceClass.js

with this diff:

h t t p s : // github.com/juce/xcs-tweaks/blob/master/xcs-devices-patch.diff


This worked for us, and i'm hoping it will help others too.

Oh my gosh, this sounds fantastic. I'll give it a shot today, as I noticed that we had the missing simulator problem mysteriously show up over the weekend for some builds, then mysteriously fixed itself. And the chain of problems you mentioned, including the problems when trying to edit a bot... yes to all of that. We're seeing the exact same set of issues. The problem is quite intermitent, so it will take some time to know if the problem is "fixed" for us or not. I'll let you know my findings.

Thanks a lot @ajouline!

Your workaround did solve a lot of problems we had. The load problem described here is gone and so are the problems i've described in this topic. I hope Apple will provide an actual and proper solution soon.


The whole device handling done by the Xcode Server currently seems to be messed up though.

E.g. the device/all-devices view in our CouchDB shows 67203 documents, i.e. there are 67203 entries for all simulator device combinations! Which seems to be way more than it's supposed to be.

great to hear that the "fix" helped.


I think you are exactly right: the root cause of all this is very likely the bloated device/all-devices view. It absolutely should not be so large. In fact, if you query the Xcode API for the list of devices, the resulting JSON is less 100kb ( h t t p s : / / 127.0.0.1:20343/api/devices ) - My guess is that the CouchDB view somehow gets a lot of duplicate entries, and then API code filters them out to present the correct list of devices, which in my case contains only 61 entries, and even if you have all available simulators installed - it'll be probably < 300. ( Certainly not 67 thousand! ) . So, i think the real proper fix would be for Apple to correct the logic that manages that CouchDB view. And then everything else will magically work again 🙂

So far so good with this "fix." Really appreciate your discovery and patch; saved my skin!

I manually cleared the all-devices view, ran list-simulators again and the number of documents went down from 67000 to 73. I had to reconfigure the (simulator) devices used by all bots afterwards because they showed 0 selected devices.

But after that all problems were gone. list-simulators now instantly returns with a proper list, the bot editing view shows all selectable devices without a significant delay and the beam.smp CPU load problems are gone as well.


My guess is that Apple somehow messed up the creation of device entries in the CouchDB database in the past causing a lot of duplicate entries. And some parts of the node code is simply not able to handle that many device entries causing all the trouble we've experienced.


Edit: we of course don't know if the bug causing duplicate entries is actually gone. I'll keep an eye on the number of entries in the all-devices view. It might just be a matter of time before the problems occur again.

Nice work.

I'm curious to see what happens to that view in CouchDB over time.

About 3 days later: there are now 3957 documents in this CouchDB view.

Seems the current Xcode Server version is a ticking time bomb. It's just a matter of time until the number of entries exceeds a certain threshold causing all the mentioned problems again.


I seriously hope Apple is going to fix this nasty bug very soon.

11449 entries today ... definitely a ticking time bomb.

This is a real nightmare. We have to clean the CouchDB at least once a week because of this bug.


Please Apple, DO STH! There isn't even a beta of a new Xcode version - which means that a possible fix is weeks if not months away.

Than you so much.

supposedly Xcode 8.3.3 has a fix for excessive CPU usage (from release notes).

But, i'm not having much luck with it...


Curious, if you or anyone else tried 8.3.3 and what the experience is so far.