testmanagerd crashing intermittently in CI builds via Jenkins & Fastlane. Why?

Our fastlane Xcode builds are failing intermittently when run by Jenkins. About 10% of builds, while running test targets, we'll get a failure with:

Code Block
[2020-11-16T16:12:10.276Z] 2020-11-16 08:11:47.435 xcodebuild[72478:2549553] [MT] IDETestOperationsObserverDebug: 415.317 elapsed -- Testing started completed.
[2020-11-16T16:12:10.276Z] 2020-11-16 08:11:47.435 xcodebuild[72478:2549553] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
[2020-11-16T16:12:10.276Z] 2020-11-16 08:11:47.435 xcodebuild[72478:2549553] [MT] IDETestOperationsObserverDebug: 415.317 sec, +415.317 sec -- end
[2020-11-16T16:12:10.276Z] Testing failed:
[2020-11-16T16:12:10.276Z] REDACTED_iPhoneTests:
[2020-11-16T16:12:10.276Z] REDACTED.app encountered an error (Failed to install or launch the test runner. If you believe this error represents a bug, please attach the result bundle at /tmp/build/jenkins-mobile-ios-develop-1107/ios/fastlane/output/derivedData/Logs/Test/Test-iPhoneSnapshotTests-2020.11.16_08-04-51--0800.xcresult. (Underlying Error: The request to open "com.REDACTED.REDACTEDapp" failed. The request was denied by service delegate (SBMainWorkspace) for reason: Busy ("Application cannot be launched because it has outstanding termination assertions"). (Underlying Error: The operation couldn’t be completed. Application cannot be launched because it has outstanding termination assertions.)))


In the simulator device logs we have a lot of "failed to bootstrap path" errors, where the report indicates that some path under Xcode-12.app cannot be found (even though in reality, it's there):

Code Block
Failed to bootstrap path: path = /Applications/Xcode-12.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/dyld_sim, error = 2: No such file or directory
Nov 16 08:11:44 Mac-169 com.apple.CoreSimulator.SimDevice.4A33E878-D3F6-4EE4-AB53-7ACD59E0B8A5[73117] (com.apple.xpc.launchd.domain.pid.testmanagerd.73149): Failed to bootstrap path: path = /Applications/Xcode-12.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit, error = 2: No such file or directory
Nov 16 08:11:44 Mac-169 com.apple.CoreSimulator.SimDevice.4A33E878-D3F6-4EE4-AB53-7ACD59E0B8A5[73117] (com.apple.xpc.launchd.domain.pid.LogArchiveCollector.73942): Failed to bootstrap path: path = /Applications/Xcode-12.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/dyld_sim, error = 2: No such file or directory
Nov 16 08:11:44 Mac-169 com.apple.CoreSimulator.SimDevice.4A33E878-D3F6-4EE4-AB53-7ACD59E0B8A5[73117] (com.apple.xpc.launchd.domain.pid.LogArchiveCollector.73942): Failed to bootstrap path: path = /Applications/Xcode-12.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit, error = 2: No such file or directory




It seems that one of the simulator clones keeps crashing and relaunching possibly (this is from the same log as the first log above, but happens just before it. (I have removed the timestamps and inserted some separators to make it easier to read, and put this at the bottom of this post since it's the longest one):

Code Block
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
xcodebuild IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/tmp/build/jenkins-mobile-ios-develop-1107/ios/fastlane/output/derivedData/Logs/Test/Test-iPhoneSnapshotTests-2020.11.16_08-04-51--0800.xcresult/Staging/1_Test/Diagnostics/DETCADERSnapshotTests-BA58BA45-D47B-418B-8E61-80EBA8B3FAAF/DETCADERSnapshotTests-A6366A3A-D533-4145-A1A2-C487CD9787E8/Session-DETCADERSnapshotTests-2020-11-16_081142-AyG7h3.log
xcodebuild IDETestOperationsObserverDebug: (02A95C0D-288A-4F0B-BE2B-BB376A8C5A16) Beginning test session DETCADERSnapshotTests-02A95C0D-288A-4F0B-BE2B-BB376A8C5A16 at 2020-11-16 08:11:42.263 with Xcode 12A7209 on target <DVTiPhoneSimulator: 0x7fd44999f1e0> {
SimDevice: Clone 2 of iPhone 7 (8FEB1993-97B1-470B-9511-C5B4693A2634, iOS 14.0, Booted)
}
xcodebuild IDETestOperationsObserverDebug: (02A95C0D-288A-4F0B-BE2B-BB376A8C5A16) Finished requesting crash reports. Continuing with testing.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
xcodebuild IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/tmp/build/jenkins-mobile-ios-develop-1107/ios/fastlane/output/derivedData/Logs/Test/Test-iPhoneSnapshotTests-2020.11.16_08-04-51--0800.xcresult/Staging/1_Test/Diagnostics/DETCADERSnapshotTests-BA58BA45-D47B-418B-8E61-80EBA8B3FAAF/DETCADERSnapshotTests-DF0A8969-D65B-4379-8DD4-1C445437C40C/Session-DETCADERSnapshotTests-2020-11-16_081142-ojHiX7.log
xcodebuild IDETestOperationsObserverDebug: (B3B8C894-1650-4C81-955D-EB3A844A1796) Beginning test session DETCADERSnapshotTests-B3B8C894-1650-4C81-955D-EB3A844A1796 at 2020-11-16 08:11:42.522 with Xcode 12A7209 on target <DVTiPhoneSimulator: 0x7fd44999f1e0> {
SimDevice: Clone 2 of iPhone 7 (8FEB1993-97B1-470B-9511-C5B4693A2634, iOS 14.0, Booted)
}
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
xcodebuild IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/tmp/build/jenkins-mobile-ios-develop-1107/ios/fastlane/output/derivedData/Logs/Test/Test-iPhoneSnapshotTests-2020.11.16_08-04-51--0800.xcresult/Staging/1_Test/Diagnostics/DETCADERSnapshotTests-BA58BA45-D47B-418B-8E61-80EBA8B3FAAF/DETCADERSnapshotTests-30F99A37-EB4E-4E36-9581-0EB2288A3664/Session-DETCADERSnapshotTests-2020-11-16_081142-RaEqDv.log
xcodebuild IDETestOperationsObserverDebug: (60E66AC3-343C-44E6-A0F1-CF1C4404CE76) Beginning test session DETCADERSnapshotTests-60E66AC3-343C-44E6-A0F1-CF1C4404CE76 at 2020-11-16 08:11:42.526 with Xcode 12A7209 on target <DVTiPhoneSimulator: 0x7fd44999f1e0> {
SimDevice: Clone 2 of iPhone 7 (8FEB1993-97B1-470B-9511-C5B4693A2634, iOS 14.0, Booted)
}
xcodebuild IDETestOperationsObserverDebug: (B3B8C894-1650-4C81-955D-EB3A844A1796) Finished requesting crash reports. Continuing with testing.
xcodebuild IDETestOperationsObserverDebug: (60E66AC3-343C-44E6-A0F1-CF1C4404CE76) Finished requesting crash reports. Continuing with testing.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
... repeats 4 more times...

Stacktrace for testmanagerd:

Code Block
Process: testmanagerd [72950]
Path: /Applications/Xcode-12.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/libexec/testmanagerd
Identifier: testmanagerd
Version: 17155
Code Type: X86-64 (Native)
Parent Process: launchd_sim [72917]
Responsible: SimulatorTrampoline [12382]
User ID: 501
Date/Time: 2020-11-16 08:12:07.926 -0800
OS Version: Mac OS X 10.15.7 (19H15)
Report Version: 12
Anonymous UUID: 5CA872E8-AE72-5688-477B-B3E543033B38
Time Awake Since Boot: 310000 seconds
System Integrity Protection: enabled
Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Exception Type: EXC_CRASH (SIGKILL)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY
Termination Reason: Namespace LIBXPC, Code 0x4
Application Specific Information:
CoreSimulator 732.17 - Device: Clone 3 of iPhone 7 (229EEFDE-E207-4409-AAC8-B00A256DC6D1) - Runtime: iOS 14.0 (18A372) - DeviceType: iPhone 7
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x00007fff5dc9fdfa mach_msg_trap + 10
1 libsystem_kernel.dylib 0x00007fff5dca0170 mach_msg + 60
2 com.apple.CoreFoundation 0x00007fff203a7e77 CFRunLoopServiceMachPort + 316
3 com.apple.CoreFoundation 0x00007fff203a258d CFRunLoopRun + 1284
4 com.apple.CoreFoundation 0x00007fff203a1b9e CFRunLoopRunSpecific + 567
5 com.apple.CoreFoundation 0x00007fff203a2cf1 CFRunLoopRun + 40
6 testmanagerd 0x000000010bcd8180 -[XCTDaemon run] + 399
7 testmanagerd 0x000000010bcb14fa main + 318
8 libdyld.dylib 0x00007fff20257415 start + 1
Thread 1:
0 libsystem_pthread.dylib 0x00007fff5dcd7b68 start_wqthread + 0
Thread 2:
0 libsystem_pthread.dylib 0x00007fff5dcd7b68 start_wqthread + 0
Thread 3:
0 libsystem_pthread.dylib 0x00007fff5dcd7b68 start_wqthread + 0
Thread 0 crashed with X86 Thread State (64-bit):
rax: 0x0000000010004005 rbx: 0x0000000007000806 rcx: 0x00007ffee3f4e638 rdx: 0x0000000000000000
rdi: 0x00007ffee3f4e820 rsi: 0x0000000007000806 rbp: 0x00007ffee3f4e690 rsp: 0x00007ffee3f4e638
r8: 0x0000000000001d03 r9: 0x00000000ffffffff r10: 0x0000000000000c00 r11: 0x0000000000000206
r12: 0x0000000007000806 r13: 0x0000000000000c00 r14: 0x00007ffee3f4e820 r15: 0x0000000000000000
rip: 0x00007fff5dc9fdfa rfl: 0x0000000000000206 cr2: 0x00007fff207357fa
Logical CPU: 0
Error Code: 0x0100001f
Trap Number: 133
Binary Images:
0x10bcaf000 - 0x10bd00fff +testmanagerd (17155) <BA16B26A-A16E-3820-B32B-D08FCA3B4579> /Applications/Xcode-12.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/libexec/testmanagerd
0x10bd8c000 - 0x10bde450f +dyld_sim (828.4) <C06A3B9E-9C65-32D9-9362-2107F24CA75E> /Applications/Xcode-12.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/dyld_sim
0x10be36000 - 0x10be5ffff com.apple.dt.instruments.DTXConnectionServices (12.0 - 64540.117.1) <4D6EBEA5-8FDB-3273-A04D-DD85DD4E2D64> /Applications/Xcode-12.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/Developer/Library/PrivateFrameworks/DTXConnectionServices.framework/DTXConnectionServices

Could this be related to the following known issue (from Xcode 12.2 patch notes)?

Simulators may not be available when running command-line tools like simctl or xcodebuild from a non-root LaunchDaemon, or when launching as a different user from the current user (for example, with sudo or launchctl). (62188195)

If so, what's the fix?

Also what does it mean, "Simulators **may** not be available..." ?
I'm also getting this on BuddyBuild - any thoughts Apple? xcodebuild test ... plus with Xcode12.2 locally I have 3 sims loading but only 1 gets used - related?

Code Block
Process: testmanagerd [13201]
Path: /Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 14.1.simruntime/Contents/Resources/RuntimeRoot/usr/libexec/testmanagerd
Identifier: testmanagerd
Version: 17155
Code Type: X86-64 (Native)
Parent Process: ??? [2618]
Responsible: SimulatorTrampoline [2327]
User ID: 501
Date/Time: 2020-12-02 21:20:55.829 -0800
OS Version: Mac OS X 10.15.5 (19F101)
Report Version: 12
Anonymous UUID: 23374693-F36A-48E0-B53B-F41ABCE2242A
Time Awake Since Boot: 11000 seconds
System Integrity Protection: disabled
Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Exception Type: EXC_CRASH (SIGKILL)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY
Termination Reason: Namespace LIBXPC, Code 0x4
Application Specific Information:
CoreSimulator 732.18.5 - Device: iPhone 12 Pro Max (3A9F7513-2ADD-4660-81E4-EC75AA1BFA11) - Runtime: iOS 14.1 (18A8394) - DeviceType: iPhone 12 Pro Max
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x00000001068b2dfa mach_msg_trap + 10
1 libsystem_kernel.dylib 0x00000001068b3170 mach_msg + 60
2 com.apple.CoreFoundation 0x0000000104c41e77 CFRunLoopServiceMachPort + 316
3 com.apple.CoreFoundation 0x0000000104c3c58d CFRunLoopRun + 1284
4 com.apple.CoreFoundation 0x0000000104c3bb9e CFRunLoopRunSpecific + 567
5 com.apple.CoreFoundation 0x0000000104c3ccf1 CFRunLoopRun + 40
6 testmanagerd 0x0000000102357180 -[XCTDaemon run] + 399
7 testmanagerd 0x00000001023304fa main + 318
8 libdyld.dylib 0x00000001064de409 start + 1
Thread 1:
0 libsystem_pthread.dylib 0x0000000106912b68 start_wqthread + 0
Thread 2:
0 libsystem_pthread.dylib 0x0000000106912b68 start_wqthread + 0
Thread 3:
0 libsystem_pthread.dylib 0x0000000106912b68 start_wqthread + 0
Thread 4:
0 libsystem_pthread.dylib 0x0000000106912b68 start_wqthread + 0
Thread 5:
0 libsystem_pthread.dylib 0x0000000106912b68 start_wqthread + 0
Thread 0 crashed with X86 Thread State (64-bit):
rax: 0x0000000010004005 rbx: 0x0000000007000806 rcx: 0x00007ffeed8cf978 rdx: 0x0000000000000000
rdi: 0x00007ffeed8cfb60 rsi: 0x0000000007000806 rbp: 0x00007ffeed8cf9d0 rsp: 0x00007ffeed8cf978
r8: 0x0000000000002703 r9: 0x00000000ffffffff r10: 0x0000000000000c00 r11: 0x0000000000000206
r12: 0x0000000007000806 r13: 0x0000000000000c00 r14: 0x00007ffeed8cfb60 r15: 0x0000000000000000
rip: 0x00000001068b2dfa rfl: 0x0000000000000206 cr2: 0x00007f9e42171de8
Logical CPU: 0
Error Code: 0x0100001f
Trap Number: 133


I am also seeing this error

Code Block
<TIMESTAMP> REDACTED[90541:220165864] [default] Failed to open URL REDACTED://company/1035: Error Domain=FBSOpenApplicationServiceErrorDomain Code=1 "The request to open "com.REDACTED.REDACTED" failed." UserInfo={BSErrorCodeDescription=RequestDenied, NSUnderlyingError=0x600001036640 {Error Domain=FBSOpenApplicationErrorDomain Code=6 "App has outstanding termination assertions" UserInfo={BSErrorCodeDescription=Busy, NSLocalizedFailureReason=App has outstanding termination assertions}}, NSLocalizedDescription=The request to open "com.REDACTED.REDACTED" failed., FBSOpenApplicationRequestID=0xbafe, NSLocalizedFailureReason=The request was denied by service delegate (SBMainWorkspace) for reason: Busy ("App has outstanding termination assertions").}


@O_G, Did you find a fix for this?
testmanagerd crashing intermittently in CI builds via Jenkins &amp; Fastlane. Why?
 
 
Q