Packet Tunnel Extension 0xdead10cc crashes on iOS 14

Hello,

We've recently started to see thousands of RUNNINGBOARD / 0xdead10cc crashes on iOS 14 from our packet tunnel extension. Based on other discussions and resources, 0xdead10cc indicates that an app is being killed for holding onto a lock during suspension. Our extension uses a SQLite database, and I do see background threads with sqlite calls in the crash logs. In a few instances of the crash logs, a background thread is handling -[NEExtensionPacketTunnelProviderContext stopWithReason:].

What does 0xdead10cc mean in the context of a network extension? Does a network extension even get suspended?
Are there scenarios where this crash would happen even when the tunnel is NOT being stopped?
Does this mean that the extension should release all locks before calling stopWithReason's completion handler?


Code Block
Exception Type: EXC_CRASH (SIGKILL)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY
Termination Reason: Namespace RUNNINGBOARD, Code 0xdead10cc
Triggered by Thread: 0
Thread 0 name:
Thread 0 Crashed:
0 libsystem_kernel.dylib 0x00000001bf38bdd0 mach_msg_trap + 8
1 libsystem_kernel.dylib 0x00000001bf38b184 mach_msg + 76 (mach_msg.c:103)
2 CoreFoundation 0x00000001933facf8 CFRunLoopServiceMachPort + 380 (CFRunLoop.c:2641)
3 CoreFoundation 0x00000001933f4ea8 CFRunLoopRun + 1216 (CFRunLoop.c:2974)
4 CoreFoundation 0x00000001933f44bc CFRunLoopRunSpecific + 600 (CFRunLoop.c:3242)
5 Foundation 0x0000000194671e30 -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 232 (NSRunLoop.m:374)
6 Foundation 0x00000001946a479c -[NSRunLoop(NSRunLoop) run] + 92 (NSRunLoop.m:399)
7 libxpc.dylib 0x00000001daa06428 _xpc_objc_main + 688 (main.m:265)
8 libxpc.dylib 0x00000001daa08700 xpc_main + 180 (init.c:1198)
9 Foundation 0x00000001946a6aa8 -[NSXPCListener resume] + 316 (NSXPCListener.m:448)
10 PlugInKit 0x00000001c1b37564 -[PKService run] + 424 (PKService.m:186)
11 PlugInKit 0x00000001c1b371a4 +[PKService main] + 576 (PKService.m:115)
12 PlugInKit 0x00000001c1b37970 +[PKService _defaultRun:arguments:] + 24 (PKService.m:233)
13 ExtensionKit 0x00000001978e5848 EXExtensionMain + 84 (EXExtensionMain.m:23)
14 Foundation 0x00000001947eeb78 NSExtensionMain + 200 (NSExtensionMain.m:13)
15 libdyld.dylib 0x00000001930bbe60 start + 4




As noted on this thread, iOS 14 is better about detecting this problem than previous systems. This probably explains why you’re only now seeing these crashes.

Does a network extension even get suspended?

It can be suspended while it’s not started.

Are there scenarios where this crash would happen even when the tunnel
is NOT being stopped?

No.

Does this mean that the extension should release all locks before
calling stopWithReason's completion handler?

Yes.

Honestly, the idea of a packet tunnel provider using SQLite strikes me as most strange.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"
Thanks for the clarifications as always!

> Does this mean that the extension should release all locks before
calling stopWithReason's completion handler?
Yes.

I'm still a bit confused by the timing since I see a few (not all) crashes with some background threads still handling stopWithReason and hasn't called the completionHandler yet. Any thoughts on what is happening there?

Code Block language
Thread 8 name:
Thread 8:
...
5 NetworkExtension 0x00000001a8115f9c -[NEExtensionPacketTunnelProviderContext stopWithReason:] + 272 (NEExtensionPacketTunnelProviderContext.m:63)
6 CoreFoundation 0x0000000197875870 invoking_ + 144
7 CoreFoundation 0x0000000197758fd0 -[NSInvocation invoke] + 300 (NSForwarding.m:3389)
8 Foundation 0x0000000198bf9d24 NSXPCCONNECTION_IS_CALLING_OUT_TO_EXPORTED_OBJECT + 20 (NSXPCConnection.m:252)
9 Foundation 0x0000000198a3787c -[NSXPCConnection _decodeAndInvokeMessageWithEvent:flags:] + 1308 (NSXPCConnection.m:776)
10 Foundation 0x0000000198bf9e78 message_handler + 224 (NSXPCConnection.m:866)
11 libxpc.dylib 0x00000001dd174454 _xpc_connection_call_event_handler + 68 (connection.c:0)
12 libxpc.dylib 0x00000001dd1747d0 _xpc_connection_mach_event + 880 (connection.c:1218)
13 libdispatch.dylib 0x00000001974aa340 _dispatch_client_callout4 + 16 (object.m:599)
14 libdispatch.dylib 0x0000000197463e10 _dispatch_mach_msg_invoke$VARIANT$mp + 384 (mach.c:2429)
15 libdispatch.dylib 0x0000000197452e70 _dispatch_lane_serial_drain$VARIANT$mp + 300 (inline_internal.h:2589)
16 libdispatch.dylib 0x0000000197464a18 _dispatch_mach_invoke$VARIANT$mp + 472 (mach.c:2751)
17 libdispatch.dylib 0x0000000197452e70 _dispatch_lane_serial_drain$VARIANT$mp + 300 (inline_internal.h:2589)
18 libdispatch.dylib 0x0000000197453ab4 _dispatch_lane_invoke$VARIANT$mp + 472 (queue.c:3862)
19 libdispatch.dylib 0x000000019745d518 _dispatch_workloop_worker_thread + 712 (queue.c:6590)
20 libsystem_pthread.dylib 0x00000001dd1545a4 _pthread_wqthread + 272 (pthread.c:2193)
21 libsystem_pthread.dylib 0x00000001dd157874 start_wqthread + 8



I see a few (not all) crashes with some background threads still
handling stopWithReason and hasn't called the completion handler
yet.

Hmmm, that stopWithReason isn’t your code but something in the system. What do frames 4 through 0 in that backtrace look like?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

What do frames 4 through 0 in that backtrace look like?

The frames are handling the stopTunnel call before the completionHandler should be called.

From a similar crash log's stack trace:

Code Block
Thread 1 name:
Thread 1:
0 <Framework> 0x0000000104e0bdc4 <Framework>PacketTunnelProvider.handleExiting(displayNotification:) + 1772 (<Framework>PacketTunnelProvider.swift:475)
1 <Framework> 0x0000000104e0b9c8 <Framework>ConfigKey.ENABLED.unsafeMutableAddressor + 36 (<Framework>ConfigKey.swift:36)
2 <Framework> 0x0000000104e0b9c8 <Framework>PacketTunnelProvider.handleExiting(displayNotification:) + 752 (<Framework>PacketTunnelProvider.swift:438)
3 <Framework> 0x0000000104e0b460 <Framework>PacketTunnelProvider.stopTunnel(with:completionHandler:) + 532 (<Framework>PacketTunnelProvider.swift:422)
4 <Framework> 0x0000000104e0b6bc @objc <Framework>PacketTunnelProvider.stopTunnel(with:completionHandler:) + 96 (<compiler-generated>:0)
5 NetworkExtension 0x0000000195258f9c -[NEExtensionPacketTunnelProviderContext stopWithReason:] + 272 (NEExtensionPacketTunnelProviderContext.m:63)
6 CoreFoundation 0x00000001849b8870 invoking_ + 144
7 CoreFoundation 0x000000018489bfd0 -[NSInvocation invoke] + 300 (NSForwarding.m:3389)
8 Foundation 0x0000000185d3cd24 NSXPCCONNECTION_IS_CALLING_OUT_TO_EXPORTED_OBJECT + 20 (NSXPCConnection.m:252)
9 Foundation 0x0000000185b7a87c -[NSXPCConnection _decodeAndInvokeMessageWithEvent:flags:] + 1308 (NSXPCConnection.m:776)
10 Foundation 0x0000000185d3ce78 message_handler + 224 (NSXPCConnection.m:866)
11 libxpc.dylib 0x00000001ca27c454 _xpc_connection_call_event_handler + 68 (connection.c:0)
12 libxpc.dylib 0x00000001ca27c7d0 _xpc_connection_mach_event + 880 (connection.c:1218)
13 libdispatch.dylib 0x00000001845ed340 _dispatch_client_callout4 + 16 (object.m:599)
14 libdispatch.dylib 0x00000001845a6e10 _dispatch_mach_msg_invoke$VARIANT$mp + 384 (mach.c:2429)
15 libdispatch.dylib 0x0000000184595e70 _dispatch_lane_serial_drain$VARIANT$mp + 300 (inline_internal.h:2589)
16 libdispatch.dylib 0x00000001845a7a18 _dispatch_mach_invoke$VARIANT$mp + 472 (mach.c:2751)
17 libdispatch.dylib 0x0000000184595e70 _dispatch_lane_serial_drain$VARIANT$mp + 300 (inline_internal.h:2589)
18 libdispatch.dylib 0x0000000184596ab4 _dispatch_lane_invoke$VARIANT$mp + 472 (queue.c:3862)
19 libdispatch.dylib 0x00000001845a0518 _dispatch_workloop_worker_thread + 712 (queue.c:6590)
20 libsystem_pthread.dylib 0x00000001ca25c5a4 _pthread_wqthread + 272 (pthread.c:2193)
21 libsystem_pthread.dylib 0x00000001ca25f874 start_wqthread + 8



Is there any chance that frame 0 (PacketTunnelProvider.swift:475) is blocked for extended periods of time?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

Is there any chance that frame 0 (PacketTunnelProvider.swift:475) is blocked for extended periods of time?

Ah... it might. How much time does an extension have to call the completionHandler? Not seeing any times in https://developer.apple.com/documentation/networkextension/nepackettunnelprovider/1406192-stoptunnel
One additional question that came to mind was what the crash logs would look like if there were no open locks but the extension blocked stopWithReason long enough.

I do see other crash logs with a similar stopWithReason stack trace that I've previous posted but with a different exception type/code. Does this indicate the blocking issue?

Code Block
Exception Type: EXC_BREAKPOINT (SIGTRAP)
Exception Codes: 0x0000000000000001, 0x00000001047dfdc4
Termination Signal: Trace/BPT trap: 5
Termination Reason: Namespace SIGNAL, Code 0x5
Terminating Process: exc handler [14635]
Triggered by Thread: 1

How much time does an extension have to call the completionHandler?

There’s no documented limit but my experience is that the system won’t let an appex run forever. The sort of timeout I’d expect is measures in at least seconds, more likely tens of seconds, so when I wrote “extended” I really meant it.

Can you post an example crash record for this 0xdead10cc crash? Not just a snippet, but the whole report. Use the text attachment feature to avoid clogging up the timeline.



I do see other crash logs with a similar stopWithReason stack trace
that I've previous posted but with a different exception type/code.
Does this indicate the blocking issue?

SIGTRAP usually means you’ve hit a Swift runtime trap, like force unwrapping a nil value or accessing an array out of bounds. That’s clearly a worry.

See Identifying the Cause of Common Crashes for more on these exception codes.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

Can you post an example crash record for this 0xdead10cc crash? Not
just a snippet, but the whole report. Use the text attachment feature
to avoid clogging up the timeline.

Please see:



SIGTRAP usually means you’ve hit a Swift runtime trap, like force unwrapping a nil value or accessing an array out of bounds. That’s clearly a worry.

Thanks!

Please see:

Thanks.

Any chance you can symbolicate this? You don’t need to worry about the Apple symbols — I can take care of them — but it’d be nice to see your symbols for context.

For symbolication help, see Adding Identifiable Symbol Names to a Crash Report.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"
Here's the symbolicated version:


Any updates or ideas on what's happening in the stack by any chance?
Thanks for the symbolicated log. Threads 4 and 8 make it clear that your provider is very involved in SQLite transactions at the time of termination. The interesting thing is thread 1, which shows that you’re definitely still within your stopTunnel(with:completionHandler:) method. There’s only two possibilities I see for being suspended with that backtrace:
  • Your stopTunnel(with:completionHandler:) implementation is taking ridiculously long amounts of time, in which case the system may just give up and suspend you anyway.

  • You have already called the completion handler but haven’t yet returned from the stopTunnel(with:completionHandler:). Once you call the completion handler, the system is free to suspend you.

If you’ve ruled out both of those possibilities, I think it’s worthwhile you filing a bug about this. Ideally your bug report would include a sysdiagnose log but, even if you can’t get that, I think it’s still worthwhile filing this based on the number of crash reports you’re getting.

Please post your bug number, just for the record.

Beyond that, I don’t have any great suggestions. The next step in debugging this would be to look at the sysdiagnose log but I realise that getting such a log will be challenging given that it’s hard to reproduce the problem.

Alternatively, you could try a workaround even without a full understanding of the problem (that’s not something I like suggesting but here it might be justified). That would involve tweaking your APMSqliteStore class to prevent the extension from suspending while you have the SQLite database open. You can do this using performExpiringActivity(withReason:using:) (remember that UIApplication background tasks are not available in an appex).

My concern here is about performance. Opening and closing the database for each transaction is likely to be expensive. If you only do this infrequently, that won’t be a problem. However, if you frequently update this database then this workaround may not be viable.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"
Packet Tunnel Extension 0xdead10cc crashes on iOS 14
 
 
Q