testmanagerd crashing intermittently in CI builds via Jenkins & Fastlane. Why?

Developer Tools & Services Xcode
O_G OP
Created Nov ’20
Replies 5
Boosts 0
Views 2.5k
Participants 3
Our fastlane Xcode builds are failing intermittently when run by Jenkins. About 10% of builds, while running test targets, we'll get a failure with:

Code Block 
[2020-11-16T16:12:10.276Z] 2020-11-16 08:11:47.435 xcodebuild[72478:2549553] [MT] IDETestOperationsObserverDebug: 415.317 elapsed -- Testing started completed.
[2020-11-16T16:12:10.276Z] 2020-11-16 08:11:47.435 xcodebuild[72478:2549553] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
[2020-11-16T16:12:10.276Z] 2020-11-16 08:11:47.435 xcodebuild[72478:2549553] [MT] IDETestOperationsObserverDebug: 415.317 sec, +415.317 sec -- end
[2020-11-16T16:12:10.276Z] Testing failed:
[2020-11-16T16:12:10.276Z] 	REDACTED_iPhoneTests:
[2020-11-16T16:12:10.276Z] 		REDACTED.app encountered an error (Failed to install or launch the test runner. If you believe this error represents a bug, please attach the result bundle at /tmp/build/jenkins-mobile-ios-develop-1107/ios/fastlane/output/derivedData/Logs/Test/Test-iPhoneSnapshotTests-2020.11.16_08-04-51--0800.xcresult. (Underlying Error: The request to open "com.REDACTED.REDACTEDapp" failed. The request was denied by service delegate (SBMainWorkspace) for reason: Busy ("Application cannot be launched because it has outstanding termination assertions"). (Underlying Error: The operation couldn’t be completed. Application cannot be launched because it has outstanding termination assertions.)))


In the simulator device logs we have a lot of "failed to bootstrap path" errors, where the report indicates that some path under Xcode-12.app cannot be found (even though in reality, it's there):

Code Block 
Failed to bootstrap path: path = /Applications/Xcode-12.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/dyld_sim, error = 2: No such file or directory
Nov 16 08:11:44 Mac-169 com.apple.CoreSimulator.SimDevice.4A33E878-D3F6-4EE4-AB53-7ACD59E0B8A5[73117] (com.apple.xpc.launchd.domain.pid.testmanagerd.73149): Failed to bootstrap path: path = /Applications/Xcode-12.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit, error = 2: No such file or directory
Nov 16 08:11:44 Mac-169 com.apple.CoreSimulator.SimDevice.4A33E878-D3F6-4EE4-AB53-7ACD59E0B8A5[73117] (com.apple.xpc.launchd.domain.pid.LogArchiveCollector.73942): Failed to bootstrap path: path = /Applications/Xcode-12.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/dyld_sim, error = 2: No such file or directory
Nov 16 08:11:44 Mac-169 com.apple.CoreSimulator.SimDevice.4A33E878-D3F6-4EE4-AB53-7ACD59E0B8A5[73117] (com.apple.xpc.launchd.domain.pid.LogArchiveCollector.73942): Failed to bootstrap path: path = /Applications/Xcode-12.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit, error = 2: No such file or directory

Replies  5
Boosts  0
Views  2.5k
Participants  3
O_G OP
Nov ’20



It seems that one of the simulator clones keeps crashing and relaunching possibly (this is from the same log as the first log above, but happens just before it. (I have removed the timestamps and inserted some separators to make it easier to read, and put this at the bottom of this post since it's the longest one):

Code Block 
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
xcodebuild  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/tmp/build/jenkins-mobile-ios-develop-1107/ios/fastlane/output/derivedData/Logs/Test/Test-iPhoneSnapshotTests-2020.11.16_08-04-51--0800.xcresult/Staging/1_Test/Diagnostics/DETCADERSnapshotTests-BA58BA45-D47B-418B-8E61-80EBA8B3FAAF/DETCADERSnapshotTests-A6366A3A-D533-4145-A1A2-C487CD9787E8/Session-DETCADERSnapshotTests-2020-11-16_081142-AyG7h3.log
xcodebuild IDETestOperationsObserverDebug: (02A95C0D-288A-4F0B-BE2B-BB376A8C5A16) Beginning test session DETCADERSnapshotTests-02A95C0D-288A-4F0B-BE2B-BB376A8C5A16 at 2020-11-16 08:11:42.263 with Xcode 12A7209 on target <DVTiPhoneSimulator: 0x7fd44999f1e0> {
		SimDevice: Clone 2 of iPhone 7 (8FEB1993-97B1-470B-9511-C5B4693A2634, iOS 14.0, Booted)
}
xcodebuild IDETestOperationsObserverDebug: (02A95C0D-288A-4F0B-BE2B-BB376A8C5A16) Finished requesting crash reports. Continuing with testing.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
xcodebuild  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/tmp/build/jenkins-mobile-ios-develop-1107/ios/fastlane/output/derivedData/Logs/Test/Test-iPhoneSnapshotTests-2020.11.16_08-04-51--0800.xcresult/Staging/1_Test/Diagnostics/DETCADERSnapshotTests-BA58BA45-D47B-418B-8E61-80EBA8B3FAAF/DETCADERSnapshotTests-DF0A8969-D65B-4379-8DD4-1C445437C40C/Session-DETCADERSnapshotTests-2020-11-16_081142-ojHiX7.log
xcodebuild IDETestOperationsObserverDebug: (B3B8C894-1650-4C81-955D-EB3A844A1796) Beginning test session DETCADERSnapshotTests-B3B8C894-1650-4C81-955D-EB3A844A1796 at 2020-11-16 08:11:42.522 with Xcode 12A7209 on target <DVTiPhoneSimulator: 0x7fd44999f1e0> {
		SimDevice: Clone 2 of iPhone 7 (8FEB1993-97B1-470B-9511-C5B4693A2634, iOS 14.0, Booted)
}
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
xcodebuild  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/tmp/build/jenkins-mobile-ios-develop-1107/ios/fastlane/output/derivedData/Logs/Test/Test-iPhoneSnapshotTests-2020.11.16_08-04-51--0800.xcresult/Staging/1_Test/Diagnostics/DETCADERSnapshotTests-BA58BA45-D47B-418B-8E61-80EBA8B3FAAF/DETCADERSnapshotTests-30F99A37-EB4E-4E36-9581-0EB2288A3664/Session-DETCADERSnapshotTests-2020-11-16_081142-RaEqDv.log
xcodebuild IDETestOperationsObserverDebug: (60E66AC3-343C-44E6-A0F1-CF1C4404CE76) Beginning test session DETCADERSnapshotTests-60E66AC3-343C-44E6-A0F1-CF1C4404CE76 at 2020-11-16 08:11:42.526 with Xcode 12A7209 on target <DVTiPhoneSimulator: 0x7fd44999f1e0> {
		SimDevice: Clone 2 of iPhone 7 (8FEB1993-97B1-470B-9511-C5B4693A2634, iOS 14.0, Booted)
}
xcodebuild IDETestOperationsObserverDebug: (B3B8C894-1650-4C81-955D-EB3A844A1796) Finished requesting crash reports. Continuing with testing.
xcodebuild IDETestOperationsObserverDebug: (60E66AC3-343C-44E6-A0F1-CF1C4404CE76) Finished requesting crash reports. Continuing with testing.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
... repeats 4 more times...

0
O_G OP
Nov ’20
Stacktrace for testmanagerd:

Code Block 
Process:               testmanagerd [72950]
Path:                  /Applications/Xcode-12.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/libexec/testmanagerd
Identifier:            testmanagerd
Version:               17155
Code Type:             X86-64 (Native)
Parent Process:        launchd_sim [72917]
Responsible:           SimulatorTrampoline [12382]
User ID:               501
Date/Time:             2020-11-16 08:12:07.926 -0800
OS Version:            Mac OS X 10.15.7 (19H15)
Report Version:        12
Anonymous UUID:        5CA872E8-AE72-5688-477B-B3E543033B38
Time Awake Since Boot: 310000 seconds
System Integrity Protection: enabled
Crashed Thread:        0  Dispatch queue: com.apple.main-thread
Exception Type:        EXC_CRASH (SIGKILL)
Exception Codes:       0x0000000000000000, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY
Termination Reason:    Namespace LIBXPC, Code 0x4
Application Specific Information:
CoreSimulator 732.17 - Device: Clone 3 of iPhone 7 (229EEFDE-E207-4409-AAC8-B00A256DC6D1) - Runtime: iOS 14.0 (18A372) - DeviceType: iPhone 7
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib        	0x00007fff5dc9fdfa mach_msg_trap + 10
1   libsystem_kernel.dylib        	0x00007fff5dca0170 mach_msg + 60
2   com.apple.CoreFoundation      	0x00007fff203a7e77 CFRunLoopServiceMachPort + 316
3   com.apple.CoreFoundation      	0x00007fff203a258d CFRunLoopRun + 1284
4   com.apple.CoreFoundation      	0x00007fff203a1b9e CFRunLoopRunSpecific + 567
5   com.apple.CoreFoundation      	0x00007fff203a2cf1 CFRunLoopRun + 40
6   testmanagerd                  	0x000000010bcd8180 -[XCTDaemon run] + 399
7   testmanagerd                  	0x000000010bcb14fa main + 318
8   libdyld.dylib                 	0x00007fff20257415 start + 1
Thread 1:
0   libsystem_pthread.dylib       	0x00007fff5dcd7b68 start_wqthread + 0
Thread 2:
0   libsystem_pthread.dylib       	0x00007fff5dcd7b68 start_wqthread + 0
Thread 3:
0   libsystem_pthread.dylib       	0x00007fff5dcd7b68 start_wqthread + 0
Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x0000000010004005  rbx: 0x0000000007000806  rcx: 0x00007ffee3f4e638  rdx: 0x0000000000000000
  rdi: 0x00007ffee3f4e820  rsi: 0x0000000007000806  rbp: 0x00007ffee3f4e690  rsp: 0x00007ffee3f4e638
   r8: 0x0000000000001d03   r9: 0x00000000ffffffff  r10: 0x0000000000000c00  r11: 0x0000000000000206
  r12: 0x0000000007000806  r13: 0x0000000000000c00  r14: 0x00007ffee3f4e820  r15: 0x0000000000000000
  rip: 0x00007fff5dc9fdfa  rfl: 0x0000000000000206  cr2: 0x00007fff207357fa
  
Logical CPU:     0
Error Code:      0x0100001f
Trap Number:     133
Binary Images:
       0x10bcaf000 -        0x10bd00fff +testmanagerd (17155) <BA16B26A-A16E-3820-B32B-D08FCA3B4579> /Applications/Xcode-12.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/libexec/testmanagerd
       0x10bd8c000 -        0x10bde450f +dyld_sim (828.4) <C06A3B9E-9C65-32D9-9362-2107F24CA75E> /Applications/Xcode-12.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/dyld_sim
       0x10be36000 -        0x10be5ffff  com.apple.dt.instruments.DTXConnectionServices (12.0 - 64540.117.1) <4D6EBEA5-8FDB-3273-A04D-DD85DD4E2D64> /Applications/Xcode-12.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/Developer/Library/PrivateFrameworks/DTXConnectionServices.framework/DTXConnectionServices

1
O_G OP
Nov ’20
Could this be related to the following known issue (from Xcode 12.2 patch notes)?

Simulators may not be available when running command-line tools like simctl or xcodebuild from a non-root LaunchDaemon, or when launching as a different user from the current user (for example, with sudo or launchctl). (62188195)

If so, what's the fix?

Also what does it mean, "Simulators **may** not be available..." ?
0
technomarknz OP
Dec ’20
I'm also getting this on BuddyBuild - any thoughts Apple? xcodebuild test ... plus with Xcode12.2 locally I have 3 sims loading but only 1 gets used - related?

Code Block 
Process:               testmanagerd [13201]
Path:                  /Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 14.1.simruntime/Contents/Resources/RuntimeRoot/usr/libexec/testmanagerd
Identifier:            testmanagerd
Version:               17155
Code Type:             X86-64 (Native)
Parent Process:        ??? [2618]
Responsible:           SimulatorTrampoline [2327]
User ID:               501
Date/Time:             2020-12-02 21:20:55.829 -0800
OS Version:            Mac OS X 10.15.5 (19F101)
Report Version:        12
Anonymous UUID:        23374693-F36A-48E0-B53B-F41ABCE2242A
Time Awake Since Boot: 11000 seconds
System Integrity Protection: disabled
Crashed Thread:        0  Dispatch queue: com.apple.main-thread
Exception Type:        EXC_CRASH (SIGKILL)
Exception Codes:       0x0000000000000000, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY
Termination Reason:    Namespace LIBXPC, Code 0x4
Application Specific Information:
CoreSimulator 732.18.5 - Device: iPhone 12 Pro Max (3A9F7513-2ADD-4660-81E4-EC75AA1BFA11) - Runtime: iOS 14.1 (18A8394) - DeviceType: iPhone 12 Pro Max
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib        	0x00000001068b2dfa mach_msg_trap + 10
1   libsystem_kernel.dylib        	0x00000001068b3170 mach_msg + 60
2   com.apple.CoreFoundation      	0x0000000104c41e77 CFRunLoopServiceMachPort + 316
3   com.apple.CoreFoundation      	0x0000000104c3c58d CFRunLoopRun + 1284
4   com.apple.CoreFoundation      	0x0000000104c3bb9e CFRunLoopRunSpecific + 567
5   com.apple.CoreFoundation      	0x0000000104c3ccf1 CFRunLoopRun + 40
6   testmanagerd                  	0x0000000102357180 -[XCTDaemon run] + 399
7   testmanagerd                  	0x00000001023304fa main + 318
8   libdyld.dylib                 	0x00000001064de409 start + 1
Thread 1:
0   libsystem_pthread.dylib       	0x0000000106912b68 start_wqthread + 0
Thread 2:
0   libsystem_pthread.dylib       	0x0000000106912b68 start_wqthread + 0
Thread 3:
0   libsystem_pthread.dylib       	0x0000000106912b68 start_wqthread + 0
Thread 4:
0   libsystem_pthread.dylib       	0x0000000106912b68 start_wqthread + 0
Thread 5:
0   libsystem_pthread.dylib       	0x0000000106912b68 start_wqthread + 0
Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x0000000010004005  rbx: 0x0000000007000806  rcx: 0x00007ffeed8cf978  rdx: 0x0000000000000000
  rdi: 0x00007ffeed8cfb60  rsi: 0x0000000007000806  rbp: 0x00007ffeed8cf9d0  rsp: 0x00007ffeed8cf978
   r8: 0x0000000000002703   r9: 0x00000000ffffffff  r10: 0x0000000000000c00  r11: 0x0000000000000206
  r12: 0x0000000007000806  r13: 0x0000000000000c00  r14: 0x00007ffeed8cfb60  r15: 0x0000000000000000
  rip: 0x00000001068b2dfa  rfl: 0x0000000000000206  cr2: 0x00007f9e42171de8
  
Logical CPU:     0
Error Code:      0x0100001f
Trap Number:     133


0
rmandala OP
Feb ’21
I am also seeing this error

Code Block 
<TIMESTAMP> REDACTED[90541:220165864] [default] Failed to open URL REDACTED://company/1035: Error Domain=FBSOpenApplicationServiceErrorDomain Code=1 "The request to open "com.REDACTED.REDACTED" failed." UserInfo={BSErrorCodeDescription=RequestDenied, NSUnderlyingError=0x600001036640 {Error Domain=FBSOpenApplicationErrorDomain Code=6 "App has outstanding termination assertions" UserInfo={BSErrorCodeDescription=Busy, NSLocalizedFailureReason=App has outstanding termination assertions}}, NSLocalizedDescription=The request to open "com.REDACTED.REDACTED" failed., FBSOpenApplicationRequestID=0xbafe, NSLocalizedFailureReason=The request was denied by service delegate (SBMainWorkspace) for reason: Busy ("App has outstanding termination assertions").}


@O_G, Did you find a fix for this?
1
testmanagerd crashing intermittently in CI builds via Jenkins &amp; Fastlane. Why?
First post date Last post date
 
 
Q