Hello,
We've recently started to see thousands of RUNNINGBOARD / 0xdead10cc crashes on iOS 14 from our packet tunnel extension. Based on other discussions and resources, 0xdead10cc indicates that an app is being killed for holding onto a lock during suspension. Our extension uses a SQLite database, and I do see background threads with sqlite calls in the crash logs. In a few instances of the crash logs, a background thread is handling -[NEExtensionPacketTunnelProviderContext stopWithReason:].
What does 0xdead10cc mean in the context of a network extension? Does a network extension even get suspended?
Are there scenarios where this crash would happen even when the tunnel is NOT being stopped?
Does this mean that the extension should release all locks before calling stopWithReason's completion handler?
Exception Type:	EXC_CRASH (SIGKILL)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note:	EXC_CORPSE_NOTIFY
Termination Reason: Namespace RUNNINGBOARD, Code 0xdead10cc
Triggered by Thread:	0
Thread 0 name:
Thread 0 Crashed:
0	 libsystem_kernel.dylib				 0x00000001bf38bdd0 mach_msg_trap + 8
1	 libsystem_kernel.dylib				 0x00000001bf38b184 mach_msg + 76 (mach_msg.c:103)
2	 CoreFoundation								 0x00000001933facf8 __CFRunLoopServiceMachPort + 380 (CFRunLoop.c:2641)
3	 CoreFoundation								 0x00000001933f4ea8 __CFRunLoopRun + 1216 (CFRunLoop.c:2974)
4	 CoreFoundation								 0x00000001933f44bc CFRunLoopRunSpecific + 600 (CFRunLoop.c:3242)
5	 Foundation										 0x0000000194671e30 -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 232 (NSRunLoop.m:374)
6	 Foundation										 0x00000001946a479c -[NSRunLoop(NSRunLoop) run] + 92 (NSRunLoop.m:399)
7	 libxpc.dylib									 0x00000001daa06428 _xpc_objc_main + 688 (main.m:265)
8	 libxpc.dylib									 0x00000001daa08700 xpc_main + 180 (init.c:1198)
9	 Foundation										 0x00000001946a6aa8 -[NSXPCListener resume] + 316 (NSXPCListener.m:448)
10	PlugInKit										 0x00000001c1b37564 -[PKService run] + 424 (PKService.m:186)
11	PlugInKit										 0x00000001c1b371a4 +[PKService main] + 576 (PKService.m:115)
12	PlugInKit										 0x00000001c1b37970 +[PKService _defaultRun:arguments:] + 24 (PKService.m:233)
13	ExtensionKit									 0x00000001978e5848 EXExtensionMain + 84 (EXExtensionMain.m:23)
14	Foundation										 0x00000001947eeb78 NSExtensionMain + 200 (NSExtensionMain.m:13)
15	libdyld.dylib								 0x00000001930bbe60 start + 4
Post
Replies
Boosts
Views
Activity
We've started to see thousands of crashes on iOS 14.0 (a few on iOS 14.2) when our app is launched for a background session. It looks like CFNetwork is logging something.
Is this a bug in CFNetwork? Has anyone encountered a similar crash?
Exception Type:	EXC_BAD_ACCESS (SIGSEGV)
Exception Subtype: KERN_INVALID_ADDRESS at 0x0000000d466bbe9c
VM Region Info: 0xd466bbe9c is not in any region.	Bytes after previous region: 45741751965	Bytes before following region: 10629693796
			REGION TYPE											START - END						 [ VSIZE] PRT/MAX SHRMOD	REGION DETAIL
			MALLOC_NANO									 280000000-2a0000000				[512.0M] rw-/rwx SM=PRV	-->	GAP OF 0xd20000000 BYTES
			commpage (reserved)					 fc0000000-1000000000			 [	1.0G] ---/--- SM=NUL	...(unallocated)
Termination Signal: Segmentation fault: 11
Termination Reason: Namespace SIGNAL, Code 0xb
Terminating Process: exc handler [202]
Triggered by Thread:	0
Thread 0 name:
Thread 0 Crashed:
0	 libobjc.A.dylib							 0x000000019422b1c0 objc_opt_isKindOfClass + 20 (objc-object.h:260)
1	 libsystem_trace.dylib				 0x0000000195dd60a4 _os_log_fmt_flatten_object + 816 (format.m:109)
2	 libsystem_trace.dylib				 0x0000000195de2500 _os_log_impl_flatten_and_send + 1716 (format.c:799)
3	 libsystem_trace.dylib				 0x0000000195de1e34 _os_log + 152 (log.c:2214)
4	 libsystem_trace.dylib				 0x0000000195de45e0 _os_log_impl + 12 (log.c:2228)
5	 CFNetwork										 0x0000000181078dd0 -[__NSURLBackgroundSession appWasLaunchedForBackgroundSession:] + 240 (BackgroundSession.mm:0)
6	 CoreFoundation								 0x0000000180949098 CFNOTIFICATIONCENTER_IS_CALLING_OUT_TO_AN_OBSERVER + 20 (CFNotificationCenter.c:706)
7	 CoreFoundation								 0x0000000180949058 ___CFXRegistrationPost_block_invoke + 48 (CFNotificationCenter.c:173)
8	 CoreFoundation								 0x0000000180948650 _CFXRegistrationPost + 400 (CFNotificationCenter.c:198)
9	 CoreFoundation								 0x0000000180948048 _CFXNotificationPost + 696 (CFNotificationCenter.c:1071)
10	Foundation										 0x0000000181b67b20 -[NSNotificationCenter postNotificationName:object:userInfo:] + 60 (NSNotification.m:575)
11	CFNetwork										 0x000000018117bd40 +[NSURLSession _sendPendingCallbacksForSessionIdentifier:] + 64 (Session.mm:222)
12	UIKitCore										 0x000000018323e990 -[UIApplication _handleNonLaunchSpecificActions:forScene:withTransitionContext:completion:] + 2932 (UIApplication.m:10283)
13	UIKitCore										 0x0000000182750538 -[UIScene _emitSceneSettingsUpdateResponseForCompletion:afterSceneUpdateWork:] + 476 (UIScene.m:1118)
14	UIKitCore										 0x0000000182751600 -[UIScene scene:didUpdateWithDiff:transitionContext:completion:] + 220 (UIScene.m:1380)
15	UIKitCore										 0x0000000182d6ad84 -[UIApplicationSceneClientAgent scene:handleEvent:withCompletion:] + 464 (UIApplicationSceneClientAgent.m:80)
16	FrontBoardServices						 0x000000018f2654b0 -[FBSScene updater:didUpdateSettings:withDiff:transitionContext:completion:] + 456 (FBSScene.m:547)
17	FrontBoardServices						 0x000000018f28dcec __94-[FBSWorkspaceScenesClient _queue_updateScene:withSettings:diff:transitionContext:completion:]_block_invoke_2 + 124 (FBSWorkspaceScenesClient.m:533)
18	FrontBoardServices						 0x000000018f272410 -[FBSWorkspace _calloutQueue_executeCalloutFromSource:withBlock:] + 232 (FBSWorkspace.m:355)
19	FrontBoardServices						 0x000000018f28dc38 __94-[FBSWorkspaceScenesClient _queue_updateScene:withSettings:diff:transitionContext:completion:]_block_invoke + 368 (FBSWorkspaceScenesClient.m:532)
20	libdispatch.dylib						 0x000000018061f280 _dispatch_client_callout + 16 (object.m:559)
21	libdispatch.dylib						 0x00000001805c4b0c _dispatch_block_invoke_direct$VARIANT$mp + 224 (queue.c:468)
22	FrontBoardServices						 0x000000018f2b14b8 FBSSERIALQUEUE_IS_CALLING_OUT_TO_A_BLOCK + 40 (FBSSerialQueue.m:184)
23	FrontBoardServices						 0x000000018f2b1180 -[FBSSerialQueue _targetQueue_performNextIfPossible] + 404 (FBSSerialQueue.m:227)
24	FrontBoardServices						 0x000000018f2b1654 -[FBSSerialQueue _performNextFromRunLoopSource] + 28 (FBSSerialQueue.m:258)
25	CoreFoundation								 0x0000000180966240 CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION + 24 (CFRunLoop.c:1967)
26	CoreFoundation								 0x0000000180966140 __CFRunLoopDoSource0 + 204 (CFRunLoop.c:2011)
27	CoreFoundation								 0x0000000180965488 __CFRunLoopDoSources0 + 256 (CFRunLoop.c:2048)
28	CoreFoundation								 0x000000018095fa40 __CFRunLoopRun + 776 (CFRunLoop.c:2925)
29	CoreFoundation								 0x000000018095f200 CFRunLoopRunSpecific + 572 (CFRunLoop.c:3242)
30	GraphicsServices							 0x0000000196a5a598 GSEventRunModal + 160 (GSEvent.c:2259)
31	UIKitCore										 0x0000000183225004 -[UIApplication _run] + 1052 (UIApplication.m:3270)
32	UIKitCore										 0x000000018322a5d8 UIApplicationMain + 164 (UIApplication.m:4739)
33	<app>					 0x00000001004853a4 main + 68 (AppDelegate.swift:26)
34	libdyld.dylib								 0x000000018063e598 start + 4
We're running into this issue with two apps (one built using Xcode 11.0 in case it matters) that both have a packet tunnel provider extension. The packet tunnel provider extension will consistently crash only on a few devices before the NEPacketTunnelProvider class has a chance to be instantiated.Console logs of the crash on an iOS 12.1 device:debug 09:47:42.976623 +0530 NFNetworkExtension Found LSExtensionPoint <private> for identifier <private>
debug 09:47:42.979965 +0530 NFNetworkExtension -[PKPlugInCore setDictionaries:] <private>
debug 09:47:42.980335 +0530 NFNetworkExtension -[PKPlugInCore setupWithName:url:bundleInfo:info:uuid:] <private>
debug 09:47:42.982089 +0530 NFNetworkExtension Found LSExtensionPoint <private> for identifier <private>
debug 09:47:42.982419 +0530 NFNetworkExtension -[PKPlugInCore setDictionaries:] <private>
debug 09:47:42.982747 +0530 NFNetworkExtension -[PKPlugInCore resolveSDK] <private> merged with SDK
debug 09:47:42.983177 +0530 NFNetworkExtension -[PKPlugInCore infoKey:] CFBundleShortVersionString => <private>
debug 09:47:43.044194 +0530 NFNetworkExtension CFPrefsManagedSource<0x10230d490> (Domain: com.symantec.nfchild.dev.NFNetExtension, User: kCFPreferencesCurrentUser, ByHost: Yes, Container: (null), Contents Need Refresh: No) loaded: an empty base plist and no additional changes from the base plist
debug 09:47:43.044322 +0530 NFNetworkExtension CFPrefsManagedSource<0x10230d660> (Domain: com.symantec.nfchild.dev.NFNetExtension, User: kCFPreferencesAnyUser, ByHost: Yes, Container: (null), Contents Need Refresh: No) loaded: an empty base plist and no additional changes from the base plist
debug 09:47:43.044505 +0530 NFNetworkExtension CFPrefsPlistSource<0x10230db80> (Domain: com.symantec.nfchild.dev.NFNetExtension, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null), Contents Need Refresh: No) loaded: an empty base plist and no additional changes from the base plist
debug 09:47:43.044689 +0530 NFNetworkExtension CFPrefsManagedSource<0x10230df20> (Domain: kCFPreferencesAnyApplication, User: kCFPreferencesCurrentUser, ByHost: Yes, Container: (null), Contents Need Refresh: No) loaded: a new base plist and no additional changes from the base plist
debug 09:47:43.044861 +0530 NFNetworkExtension CFPrefsPlistSource<0x10230e020> (Domain: kCFPreferencesAnyApplication, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null), Contents Need Refresh: No) loaded: a new base plist and no additional changes from the base plist
debug 09:47:43.045084 +0530 NFNetworkExtension CFPrefsPlistSource<0x10230e0f0> (Domain: com.symantec.nfchild.dev.NFNetExtension, User: kCFPreferencesAnyUser, ByHost: Yes, Container: (null), Contents Need Refresh: No) loaded: an empty base plist and no additional changes from the base plist
debug 09:47:43.045727 +0530 NFNetworkExtension CFPrefsPlistSource<0x10230e1f0> (Domain: kCFPreferencesAnyApplication, User: kCFPreferencesAnyUser, ByHost: Yes, Container: (null), Contents Need Refresh: No) loaded: an empty base plist and no additional changes from the base plist
debug 09:47:43.045936 +0530 NFNetworkExtension looked up value <private> for key AppleLanguages in CFPrefsPlistSource<0x10230e020> (Domain: kCFPreferencesAnyApplication, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null), Contents Need Refresh: No) via CFPrefsSearchListSource<0x10230d230> (Domain: com.symantec.nfchild.dev.NFNetExtension, Container: (null))
debug 09:47:43.046063 +0530 NFNetworkExtension setting new value <private> for key <private> in CFPrefsSource<0x10230dfe0> (Domain: Volatile, User: , ByHost: No, Container: , Contents Need Refresh: No)
debug 09:47:43.046343 +0530 NFNetworkExtension setting new value <private> for key <private> in CFPrefsSource<0x10230dfe0> (Domain: Volatile, User: , ByHost: No, Container: , Contents Need Refresh: No)
debug 09:47:43.127001 +0530 NFNetworkExtension setting new value <private> for key <private> in CFPrefsSource<0x10230dfe0> (Domain: Volatile, User: , ByHost: No, Container: , Contents Need Refresh: No)
default 09:47:43.127668 +0530 NFNetworkExtension *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '*** -[__NSDictionaryM setObject:forKey:]: object cannot be nil (key: E9220768-902D-467B-BC97-AC1912A14ACE)'
*** First throw call stack:
(0x1c88f9ea4 0x1c7ac9a50 0x1c8871384 0x1c87e4920 0x1c933fb5c 0x1c933f1e4 0x1c83316c8 0x1c8332484 0x1c82d9bd0 0x1c82da718 0x1c82e2eb8 0x1c85150dc 0x1c8517cec)Stack trace:Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY
Triggered by Thread: 3
Last Exception Backtrace:
0 CoreFoundation 0x182f86d8c __exceptionPreprocess + 228 (NSException.m:166)
1 libobjc.A.dylib 0x1821405ec objc_exception_throw + 56 (objc-exception.mm:521)
2 CoreFoundation 0x182f1f750 _CFThrowFormattedException + 112 (CFObject.m:1946)
3 CoreFoundation 0x182e54c6c -[__NSDictionaryM setObject:forKey:] + 948 (NSDictionaryM.m:176)
4 Foundation 0x1839664f8 -[_NSExtensionContextVendor _setPrincipalObject:forUUID:] + 100 (NSExtensionContext.m:362)
5 Foundation 0x183965b84 __105-[_NSExtensionContextVendor _beginRequestWithExtensionItems:listenerEndpoint:withContextUUID... + 688 (NSExtensionContext.m:264)
6 libdispatch.dylib 0x182878aa0 _dispatch_call_block_and_release + 24 (init.c:994)
7 libdispatch.dylib 0x182878a60 _dispatch_client_callout + 16 (object.m:507)
8 libdispatch.dylib 0x1828829b4 _dispatch_queue_serial_drain$VARIANT$mp + 608 (inline_internal.h:2500)
9 libdispatch.dylib 0x1828832fc _dispatch_queue_invoke$VARIANT$mp + 336 (queue.c:5290)
10 libdispatch.dylib 0x182883cc8 _dispatch_root_queue_drain_deferred_wlh$VARIANT$mp + 340 (queue.c:5908)
11 libdispatch.dylib 0x18288c098 _dispatch_workloop_worker_thread$VARIANT$mp + 668 (source.c:2529)
12 libsystem_pthread.dylib 0x182****70 _pthread_wqthread + 860 (pthread.c:2218)
13 libsystem_pthread.dylib 0x182babb08 start_wqthread + 4
Thread 0 name:
Thread 0:
0 libsystem_kernel.dylib 0x00000001829ebde8 mach_msg_trap + 8
1 libsystem_kernel.dylib 0x00000001829ebc60 mach_msg + 72 (mach_msg.c:103)
2 CoreFoundation 0x0000000182f2ee40 __CFRunLoopServiceMachPort + 196 (CFRunLoop.c:2613)
3 CoreFoundation 0x0000000182f2c908 __CFRunLoopRun + 1568 (CFRunLoop.c:2969)
4 CoreFoundation 0x0000000182e4cda8 CFRunLoopRunSpecific + 552 (CFRunLoop.c:3245)
5 Foundation 0x00000001838c1674 -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 304 (NSRunLoop.m:367)
6 Foundation 0x00000001839361a8 -[NSRunLoop(NSRunLoop) run] + 88 (NSRunLoop.m:389)
7 libxpc.dylib 0x0000000182bf5a6c _xpc_objc_main + 516 (main.m:167)
8 libxpc.dylib 0x0000000182bf7b40 xpc_main + 180 (init.c:1476)
9 Foundation 0x0000000183af1fd4 -[NSXPCListener resume] + 180 (NSXPCListener.m:253)
10 PlugInKit 0x0000000188602ec4 -[PKService run] + 712 (PKService.m:105)
11 PlugInKit 0x0000000188602ab4 +[PKService main] + 56 (PKService.m:39)
12 PlugInKit 0x0000000188602ee8 +[PKService _defaultRun:arguments:] + 20 (PKService.m:116)
13 Foundation 0x0000000183ad8a98 NSExtensionMain + 64 (NSExtensionMain.m:23)
14 libdyld.dylib 0x00000001828ddfc0 start + 4
Thread 1:
0 libsystem_kernel.dylib 0x0000000182a0dd78 __workq_kernreturn + 8
1 libsystem_pthread.dylib 0x0000000182babeb4 _pthread_wqthread + 928 (pthread.c:0)
2 libsystem_pthread.dylib 0x0000000182babb08 start_wqthread + 4
Thread 2:
0 libsystem_pthread.dylib 0x0000000182babb04 start_wqthread + 0 (pthread.c:2458)
Thread 3 name:
Thread 3 Crashed:
0 libsystem_kernel.dylib 0x0000000182a0d2e0 __pthread_kill + 8
1 libsystem_pthread.dylib 0x0000000182bae288 pthread_kill$VARIANT$mp + 376 (pthread.c:1484)
2 libsystem_c.dylib 0x000000018297bd0c abort + 140 (abort.c:91)
3 libc++abi.dylib 0x00000001821172c8 abort_message + 132 (abort_message.cpp:75)
4 libc++abi.dylib 0x0000000182117470 default_terminate_handler() + 304 (cxa_default_handlers.cpp:68)
5 libobjc.A.dylib 0x00000001821408d4 _objc_terminate() + 124 (objc-exception.mm:657)
6 libc++abi.dylib 0x000000018213137c std::__terminate(void (*)()) + 16 (cxa_handlers.cpp:66)
7 libc++abi.dylib 0x0000000182130ccc __cxa_throw + 132 (cxa_exception.cpp:130)
8 libobjc.A.dylib 0x0000000182140720 objc_exception_throw + 364 (objc-exception.mm:547)
9 CoreFoundation 0x0000000182f1f750 _CFThrowFormattedException + 112 (CFObject.m:1946)
10 CoreFoundation 0x0000000182e54c6c -[__NSDictionaryM setObject:forKey:] + 948 (NSDictionaryM.m:176)
11 Foundation 0x00000001839664f8 -[_NSExtensionContextVendor _setPrincipalObject:forUUID:] + 100 (NSExtensionContext.m:362)
12 Foundation 0x0000000183965b84 __105-[_NSExtensionContextVendor _beginRequestWithExtensionItems:listenerEndpoint:withContextUUID... + 688 (NSExtensionContext.m:264)
13 libdispatch.dylib 0x0000000182878aa0 _dispatch_call_block_and_release + 24 (init.c:994)
14 libdispatch.dylib 0x0000000182878a60 _dispatch_client_callout + 16 (object.m:507)
15 libdispatch.dylib 0x00000001828829b4 _dispatch_queue_serial_drain$VARIANT$mp + 608 (inline_internal.h:2500)
16 libdispatch.dylib 0x00000001828832fc _dispatch_queue_invoke$VARIANT$mp + 336 (queue.c:5290)
17 libdispatch.dylib 0x0000000182883cc8 _dispatch_root_queue_drain_deferred_wlh$VARIANT$mp + 340 (queue.c:5908)
18 libdispatch.dylib 0x000000018288c098 _dispatch_workloop_worker_thread$VARIANT$mp + 668 (source.c:2529)
19 libsystem_pthread.dylib 0x0000000182****70 _pthread_wqthread + 860 (pthread.c:2218)
20 libsystem_pthread.dylib 0x0000000182babb08 start_wqthread + 4Entitlements from the PacketTunnelExtension binary using `codesign -d --entilements -- ...` show:<key>com.apple.developer.networking.networkextension</key>
<array>
<string>packet-tunnel-provider</string>
</array>Info.plist<key>NSExtensionPointIdentifier</key>
<string>com.apple.networkextension.packet-tunnel</string>
<key>NSExtensionPrincipalClass</key>
<string>$(PRODUCT_MODULE_NAME).PacketTunnelProvider</string>Any ideas on what's happening here?