CouchDB problems with Xcode CI

I wish I could help, but I'm at least here to say "Me Too!" after installing Xcode 8.3 in Xcode Server on our Late 2013 Mac Pro beam.smp sits at about 200% cpu usage and things just generally don't work. 😟

While I haven't monitored the CPU load issue, I have had the case of something going haywire with the couchdb.
I haven't been able to open the couchdb to look at it (you must be an admin to view this db, account permissions issue).
Ultimately I had to just **** away the Xcode Server instance and start from zero, which also meant recreating bots.
Recreating the bots isn't super difficult for my needs, but I do lose the integration history, which *****.

So, I'm basically adding my name to the "Me too!" list of "Xcode 8.3 + Xcode Server 5.3 = couchdb screwiness"
Does anyone know a good strategy for backing up the couch db file, so I can get a clean server up and running with old server bots quickly?

Hi,

I'm sorry you're hitting this issue. It's not something we're aware of, and we need some more information to figure it out.

Please run:

sudo xcrun xcsdiagnose

... then file a bug, attach the output from that command, and post the bug number here.

Thank you!

Brent

I've submitted mine as bug report #31551004

Thanks for the steps to help diagnose. I've also run into this issue and filed a bug. It's rdar://31556253

Thanks Brent for your answer.

We've filed a bug report with the number 31556463.

Thank you for the bug reports and diagnostics!

Are your bots configured to test on devices? If so, how many devices are attached to the server? Some of the logs suggest that the device service that Xcode Server uses to detect devices is working way too hard.

If your bots are testing on devices you might try temporarily disconnecting those devices and reconfiguring the bot to test on simulators instead. Please let us know if that makes any difference.

Thank you,

Brent

@Brent our bug ticket got closed because it was marked as a duplicate of 31556253. So does this mean you've found the cause of this problem? Can you shed any light on this?

Our bots don't use devices for testing btw.

adding my name to the "Me too" list.

We have a CI system which uses Xcode Server. Things we working quite well with many Xcode Server versions starting with 6.4 and all the way to 8.2.1, but since the switch to 8.3 - it's been a nightmare. (an update to 8.3.1 did not help). Not 100% certain, but it appears that beam.smp hugging the CPU might be the cause of it. From what i can tell it makes the API (run by node) to become unresponsive, which in turn has a cascading effect on other things like: no device simulators being found at random times, when tests need to run, bot configuration changes not saved, unable to retrieve information about integrations because of time-outs.

As a side note: /Applications/Xcode.app/Contents/Developer/usr/share/xcs/xcsd/app.js has an interesting comment about re-indexing of CouchDB database. Feels like this might be related.

Anton.

Our bots only test on two simulators: iPad Air 2 w/9.3 and iPhone 7 w/10.3

I had to downgrade Server.app to 5.2 and Xcode to 8.2.1 to get rid of this unpleasant behavior. I just use Xcode server's git part. So it is ok to downgrade.

For what it's worth, we are also blocked by this. Our entire CI process is completely pooched. XCode Server, as of the most recent version (5.3) simply doesn't work. At all.

Like others, I'm seeing:

• beam.smp going coocoo for coacoapuffs. Watching it today it even spiked up to 467% (this thing is operating on a Mac Pro).
• Bots can't seem to ever find the devices ("The devices configured for testing were not found") despite bots not using devices and only using 1-2 simulators
• We're also increasingly seeing the error message "Failed to build workspace *** with scheme yyy", despite us only ever using that scheme, always. It's our Xcode CI scheme, basically. That's never changed.

We've resorted to having to turning off tests on Xcode Server, while also still having to restart the server multiple times per day.

The current version is not only unuseful, it's actively lowering the producitvity of our entire team.

Are there any workarounds that we can do here? We are getting killed by this.

We have at least a similar problem with Server.app 5.3 and Xcode 8.3+, but instead of beam.smp it's node processes owned by _xcsd. Same results, all CI functionality is useless after a few minutes of uptime. Submitted rdar://31792343 and mirrored to https://openradar.appspot.com/radar?id=4927734979493888

Note: The solution from @ajouline below seems to be a good workaround for now!

I've got the exact same issue, almost unable to use Xcode Server anymore. ( Xcode 8.3.2 + Xcode Server 5.3 )

I'm having the same issue and seeing this with regards to beam.smp. I only have one bot and it is running on 3 simulators only.