Xcode 12.x Build Service Error: Integration results could not be uploaded.

Here's some more info from the buildService.log file. Rather than a 30 second timeout, it looks like a 10 second one due to an internal network call being made to the bot server (?)

Code Block 
Oct  7 13:28:35  [54077] <Info>: Completed integration step XCSProcessBuildResultsIntegrationStep without error
Oct  7 13:28:36  [54077] <Info>: Uploading file /Users/xcodeserver/Library/Caches/XCSBuilder/Integration-5edb0de71a3e9dd13e1844b98966e59e/xcodebuild_result.xcresult.zip
Oct  7 13:38:36  [54077] <Error>: Error: The request timed out.
Oct  7 13:38:36  [54077] <Error>: Error trying to upload /Users/xcodeserver/Library/Caches/XCSBuilder/Integration-5edb0de71a3e9dd13e1844b98966e59e/xcodebuild_result.xcresult.zip: Unable to perform the request. CFNetwork error: The request timed out. (-1001 - kCFURLErrorTimedOut)
Oct  7 13:38:36  [54077] <Error>: XCSProcessTestResultsIntegrationStep finished integration with an error: Error Domain=XCSBuildServiceDomain Code=-1 "Integration results could not be uploaded. Test attachments and integration logs may be unavailable for this integration." UserInfo={NSLocalizedDescription=Integration results could not be uploaded. Test attachments and integration logs may be unavailable for this integration.}
Oct  7 13:38:37  [54077] <Info>: Executing trigger 'post_build_error'

After observing the system in more detail, I see more of what's happening. When I initiate an integration for a bot, a unix-executable file is generated in the folder /Library/Developer/XcodeServer/IntegrationAssets. This file is supposed to have the permissions -rwxrwx--- but instead has permissions -rw-r-r--.

Before: -rw-r--r--   1 _xcsd  _xcs     48948 Oct  9 14:12 5b6890ca4afc71b3c85c5f931176d970
After: -rwxrwx---   1 _xcsd  _xcs     48948 Oct  9 14:12 5b6890ca4afc71b3c85c5f931176d970

When the bot finishes and the file permissions are broken, the bot fails and gives me the warning about not being able to upload the integration assets. It also fails to delete the folder for the integration assets located at ~/Library/Caches/XCSBuilder/Integration-04f7a1d672eaf280a992b4001d5367b4 (the guid changes per build)

If I run sudo xcrun xcscontrol --fix-permissions in time, the permissions get fixed and the bot finishes normally.

Here's the real bummer: The permissions are only wrong about 1 out of every 4 integrations.

Anybody have any further suggestions? What script is creating that file with the incorrect permissions sometimes?

Got the same annoying problem, thank you for your research!

Maybe creating an applescript or something similar could fix the problem? Watching the folder and just executing the sudo xcrun or maybe directly setting the correct permissions with sudo chmod 0770 file?

Edit: Have you filed a bug report with apple https://bugreport.apple.com ?

This permissions problem is interesting, because today I finally saw a more explicit message than just "integration results could not be uploaded."

Code Block 
Build Service Warning: “com.apple.Safari.SafeBrowsing.Service-2020-10-22-030548.diag” couldn’t be copied because you don’t have permission to access “Integration-89776b80b6693d016b6ed8b7045e7606”.
Reason: You don’t have permission.

Code Block Nov 24 04:13:47  [28884] <Info>: Completed integration step XCSProcessBuildResultsIntegrationStep without error
Nov 24 04:13:47  [28884] <Info>: Uploading file /Users/server/Library/Caches/XCSBuilder/Integration-0e9d438d371a0c194d7926a703d59b19/xcodebuild_result.xcresult.zip
Nov 24 04:23:48  [28884] <Error>: Error: The request timed out.
Nov 24 04:23:48  [28884] <Error>: Error trying to upload /Users/server/Library/Caches/XCSBuilder/Integration-0e9d438d371a0c194d7926a703d59b19/xcodebuild_result.xcresult.zip: Unable to perform the request. CFNetwork error: The request timed out. (-1001 - kCFURLErrorTimedOut)
Nov 24 04:23:48  [28884] <Error>: XCSProcessTestResultsIntegrationStep finished integration with an error: Error Domain=XCSBuildServiceDomain Code=-1 "Integration results could not be uploaded. Test attachments and integration logs may be unavailable for this integration." UserInfo={NSLocalizedDescription=Integration results could not be uploaded. Test attachments and integration logs may be unavailable for this integration.}
Nov 24 04:23:48  [28884] <Info>: Executing trigger 'testflight upload'

I'm having that exact same trouble a lot. It sometimes feels like a bot will work great at first, then slowly degrade with this error occurring more and more. If a bot exhibits this issue, it is a toin-coss if I'll ever get it to build properly.

The main problem here for me is that if this occurs, the bot's .xcarchive build phase is skipped.
So I have a critical workflow which uploads the .xcarchive to TestFlight. However, if this "failed to upload integration results" error occurs, the .xcarchive is never built. Xcode Server just skips to the first post-integration script immediately.

This has broken my workflow and puts us in a tough position for QA certification of the release builds.

Bots are created fresh with each weekly release, so "recreating the bot" is not working as a solution here.

Looking at our logs, I don't think that's a 10 second timeout. I think it is a 10 minute timeout, no?
In jerimiah's log it starts at 13:28:36 and times out at 13:38:36
hour:minute:sec format, it looks like, based on relative timings for other logs.
That exactly matches mine of 10 minutes.

And where is it "uploading" to?

I'm getting much better stability if I do the following

1. sudo xcrun xcscontrol --fix-permissions is run constantly by the maintenance script

2. It’s also run prior to doing an integration

3. Set the bot to Always Clean

4. Occasionally run Edit bot.. > Repositories > Replace Repositories… (can only be done manually it seems)

I'm going to give up, for now, in having the bot use its automatic archive generation and instead do it manually through a build script. The same bot builds its Firebase distributions flawlessly, so I guess I'll just modify that process to also do a TestFlight build. Frustrating regression...

I spoke too soon. This morning, things were moving more smoothly than before.
Tonight, I can't get Xcode Server to build an .xcarchive no matter what I try.
Fixing permissions, clearing out the caches, rebooting the server... doesn't seem to matter.
I see in the GUI "Uploading integration logs" and it's been that way for 11 minutes now.
This time, deleting the bot and rebuilding it worked... for now... it is not a guarantee, though it usually works for a while before eventually degrading.

Official bug report here: (Feedback assistant)

FB8925312 (Bots sometime fail to 'upload integration results' resulting in integration failure)

Which versions of macOS are people seeing this problem?

I was using Xcode Server 11.6 with macOS 10.15.7 without any problems. After I updated to Xcode Server 12.5, I started seeing exactly the same problems mentioned above.

I'm wondering if updating my server to macOS 11.2 could be any help in addressing these permission issues?

I wondering how you run the fix-permissions scripts from a bot. So far Xcode Server is having a 100% failure rate for me. Different errors at different times. Gotta say I love the 1" maximum feedback area in the bot results too. Best you can do is make it smaller. So much vertical whitespace. It's like being on Stack Overflow and having to constantly scroll left and right to read something while you see huge margins on the left and right. I'm thinking that that alone shows the amount of love this isn't getting, so I'm going to give up on it.