Packet Tunnel Extension 0xdead10cc crashes on iOS 14

As noted on this thread, iOS 14 is better about detecting this problem than previous systems. This probably explains why you’re only now seeing these crashes.

Does a network extension even get suspended?

It can be suspended while it’s not started.

Are there scenarios where this crash would happen even when the tunnel
is NOT being stopped?

No.

Does this mean that the extension should release all locks before
calling stopWithReason's completion handler?

Yes.

Honestly, the idea of a packet tunnel provider using SQLite strikes me as most strange.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

Thanks for the clarifications as always!

> Does this mean that the extension should release all locks before
calling stopWithReason's completion handler?
Yes.

I'm still a bit confused by the timing since I see a few (not all) crashes with some background threads still handling stopWithReason and hasn't called the completionHandler yet. Any thoughts on what is happening there?

Code Block language
Thread 8 name:
Thread 8:
...
5   NetworkExtension              	0x00000001a8115f9c -[NEExtensionPacketTunnelProviderContext stopWithReason:] + 272 (NEExtensionPacketTunnelProviderContext.m:63)
6   CoreFoundation                	0x0000000197875870 invoking_ + 144
7   CoreFoundation                	0x0000000197758fd0 -[NSInvocation invoke] + 300 (NSForwarding.m:3389)
8   Foundation                    	0x0000000198bf9d24 NSXPCCONNECTION_IS_CALLING_OUT_TO_EXPORTED_OBJECT + 20 (NSXPCConnection.m:252)
9   Foundation                    	0x0000000198a3787c -[NSXPCConnection _decodeAndInvokeMessageWithEvent:flags:] + 1308 (NSXPCConnection.m:776)
10  Foundation                    	0x0000000198bf9e78 message_handler + 224 (NSXPCConnection.m:866)
11  libxpc.dylib                  	0x00000001dd174454 _xpc_connection_call_event_handler + 68 (connection.c:0)
12  libxpc.dylib                  	0x00000001dd1747d0 _xpc_connection_mach_event + 880 (connection.c:1218)
13  libdispatch.dylib             	0x00000001974aa340 _dispatch_client_callout4 + 16 (object.m:599)
14  libdispatch.dylib             	0x0000000197463e10 _dispatch_mach_msg_invoke$VARIANT$mp + 384 (mach.c:2429)
15  libdispatch.dylib             	0x0000000197452e70 _dispatch_lane_serial_drain$VARIANT$mp + 300 (inline_internal.h:2589)
16  libdispatch.dylib             	0x0000000197464a18 _dispatch_mach_invoke$VARIANT$mp + 472 (mach.c:2751)
17  libdispatch.dylib             	0x0000000197452e70 _dispatch_lane_serial_drain$VARIANT$mp + 300 (inline_internal.h:2589)
18  libdispatch.dylib             	0x0000000197453ab4 _dispatch_lane_invoke$VARIANT$mp + 472 (queue.c:3862)
19  libdispatch.dylib             	0x000000019745d518 _dispatch_workloop_worker_thread + 712 (queue.c:6590)
20  libsystem_pthread.dylib       	0x00000001dd1545a4 _pthread_wqthread + 272 (pthread.c:2193)
21  libsystem_pthread.dylib       	0x00000001dd157874 start_wqthread + 8

I see a few (not all) crashes with some background threads still
handling stopWithReason and hasn't called the completion handler
yet.

Hmmm, that stopWithReason isn’t your code but something in the system. What do frames 4 through 0 in that backtrace look like?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

What do frames 4 through 0 in that backtrace look like?

The frames are handling the stopTunnel call before the completionHandler should be called.

From a similar crash log's stack trace:

Code Block 
Thread 1 name:
Thread 1:
0   <Framework>                          	0x0000000104e0bdc4 <Framework>PacketTunnelProvider.handleExiting(displayNotification:) + 1772 (<Framework>PacketTunnelProvider.swift:475)
1   <Framework>                          	0x0000000104e0b9c8 <Framework>ConfigKey.ENABLED.unsafeMutableAddressor + 36 (<Framework>ConfigKey.swift:36)
2   <Framework>                          	0x0000000104e0b9c8 <Framework>PacketTunnelProvider.handleExiting(displayNotification:) + 752 (<Framework>PacketTunnelProvider.swift:438)
3   <Framework>                          	0x0000000104e0b460 <Framework>PacketTunnelProvider.stopTunnel(with:completionHandler:) + 532 (<Framework>PacketTunnelProvider.swift:422)
4   <Framework>                          	0x0000000104e0b6bc @objc <Framework>PacketTunnelProvider.stopTunnel(with:completionHandler:) + 96 (<compiler-generated>:0)
5   NetworkExtension              	0x0000000195258f9c -[NEExtensionPacketTunnelProviderContext stopWithReason:] + 272 (NEExtensionPacketTunnelProviderContext.m:63)
6   CoreFoundation                	0x00000001849b8870 invoking_ + 144
7   CoreFoundation                	0x000000018489bfd0 -[NSInvocation invoke] + 300 (NSForwarding.m:3389)
8   Foundation                    	0x0000000185d3cd24 NSXPCCONNECTION_IS_CALLING_OUT_TO_EXPORTED_OBJECT + 20 (NSXPCConnection.m:252)
9   Foundation                    	0x0000000185b7a87c -[NSXPCConnection _decodeAndInvokeMessageWithEvent:flags:] + 1308 (NSXPCConnection.m:776)
10  Foundation                    	0x0000000185d3ce78 message_handler + 224 (NSXPCConnection.m:866)
11  libxpc.dylib                  	0x00000001ca27c454 _xpc_connection_call_event_handler + 68 (connection.c:0)
12  libxpc.dylib                  	0x00000001ca27c7d0 _xpc_connection_mach_event + 880 (connection.c:1218)
13  libdispatch.dylib             	0x00000001845ed340 _dispatch_client_callout4 + 16 (object.m:599)
14  libdispatch.dylib             	0x00000001845a6e10 _dispatch_mach_msg_invoke$VARIANT$mp + 384 (mach.c:2429)
15  libdispatch.dylib             	0x0000000184595e70 _dispatch_lane_serial_drain$VARIANT$mp + 300 (inline_internal.h:2589)
16  libdispatch.dylib             	0x00000001845a7a18 _dispatch_mach_invoke$VARIANT$mp + 472 (mach.c:2751)
17  libdispatch.dylib             	0x0000000184595e70 _dispatch_lane_serial_drain$VARIANT$mp + 300 (inline_internal.h:2589)
18  libdispatch.dylib             	0x0000000184596ab4 _dispatch_lane_invoke$VARIANT$mp + 472 (queue.c:3862)
19  libdispatch.dylib             	0x00000001845a0518 _dispatch_workloop_worker_thread + 712 (queue.c:6590)
20  libsystem_pthread.dylib       	0x00000001ca25c5a4 _pthread_wqthread + 272 (pthread.c:2193)
21  libsystem_pthread.dylib       	0x00000001ca25f874 start_wqthread + 8

Is there any chance that frame 0 (PacketTunnelProvider.swift:475) is blocked for extended periods of time?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

Is there any chance that frame 0 (PacketTunnelProvider.swift:475) is blocked for extended periods of time?

Ah... it might. How much time does an extension have to call the completionHandler? Not seeing any times in https://developer.apple.com/documentation/networkextension/nepackettunnelprovider/1406192-stoptunnel

One additional question that came to mind was what the crash logs would look like if there were no open locks but the extension blocked stopWithReason long enough.

I do see other crash logs with a similar stopWithReason stack trace that I've previous posted but with a different exception type/code. Does this indicate the blocking issue?

Code Block 
Exception Type:  EXC_BREAKPOINT (SIGTRAP)
Exception Codes: 0x0000000000000001, 0x00000001047dfdc4
Termination Signal: Trace/BPT trap: 5
Termination Reason: Namespace SIGNAL, Code 0x5
Terminating Process: exc handler [14635]
Triggered by Thread:  1

How much time does an extension have to call the completionHandler?

There’s no documented limit but my experience is that the system won’t let an appex run forever. The sort of timeout I’d expect is measures in at least seconds, more likely tens of seconds, so when I wrote “extended” I really meant it.

Can you post an example crash record for this 0xdead10cc crash? Not just a snippet, but the whole report. Use the text attachment feature to avoid clogging up the timeline.

---

I do see other crash logs with a similar stopWithReason stack trace
that I've previous posted but with a different exception type/code.
Does this indicate the blocking issue?

SIGTRAP usually means you’ve hit a Swift runtime trap, like force unwrapping a nil value or accessing an array out of bounds. That’s clearly a worry.

See Identifying the Cause of Common Crashes for more on these exception codes.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

Can you post an example crash record for this 0xdead10cc crash? Not
just a snippet, but the whole report. Use the text attachment feature
to avoid clogging up the timeline.

Please see:

SIGTRAP usually means you’ve hit a Swift runtime trap, like force unwrapping a nil value or accessing an array out of bounds. That’s clearly a worry.

Thanks!

Please see:

Thanks.

Any chance you can symbolicate this? You don’t need to worry about the Apple symbols — I can take care of them — but it’d be nice to see your symbols for context.

For symbolication help, see Adding Identifiable Symbol Names to a Crash Report.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

Here's the symbolicated version:

Any updates or ideas on what's happening in the stack by any chance?

Thanks for the symbolicated log. Threads 4 and 8 make it clear that your provider is very involved in SQLite transactions at the time of termination. The interesting thing is thread 1, which shows that you’re definitely still within your stopTunnel(with:completionHandler:) method. There’s only two possibilities I see for being suspended with that backtrace:

• Your stopTunnel(with:completionHandler:) implementation is taking ridiculously long amounts of time, in which case the system may just give up and suspend you anyway.

• You have already called the completion handler but haven’t yet returned from the stopTunnel(with:completionHandler:). Once you call the completion handler, the system is free to suspend you.

If you’ve ruled out both of those possibilities, I think it’s worthwhile you filing a bug about this. Ideally your bug report would include a sysdiagnose log but, even if you can’t get that, I think it’s still worthwhile filing this based on the number of crash reports you’re getting.

Please post your bug number, just for the record.

Beyond that, I don’t have any great suggestions. The next step in debugging this would be to look at the sysdiagnose log but I realise that getting such a log will be challenging given that it’s hard to reproduce the problem.

Alternatively, you could try a workaround even without a full understanding of the problem (that’s not something I like suggesting but here it might be justified). That would involve tweaking your APMSqliteStore class to prevent the extension from suspending while you have the SQLite database open. You can do this using performExpiringActivity(withReason:using:) (remember that UIApplication background tasks are not available in an appex).

My concern here is about performance. Opening and closing the database for each transaction is likely to be expensive. If you only do this infrequently, that won’t be a problem. However, if you frequently update this database then this workaround may not be viable.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"