CFNetwork crash in CFURLRequestSetMainDocumentURL

I have received two strange crash reports …

Can you repost these crash reports but this time do the redaction as described in Posting a Crash Report? Your current redaction approach is causing difficulties at my end.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Attached to this post are crash logs with better redaction / alignment.

This is probably worthy of a separate topic, but I do briefly want to mention that the iPadOS 15.2 crash log failed to symbolicate automatically, and is still not fully symbolicated. I had to manually download iOS 15.2 arm64 and arm64e symbols and run symbolication with the symbolicatecrash command line utility. Xcode 13.2 freezes when I try to symbolicate the crash log through the Devices and Simulators window, and apparently Xcode 13.2 did not include the 15.2 symbols. Perhaps this is a result of downloading Xcode 13.2 manually from the Apple Developer downloads website, which was required due to the App Store Xcode issue described here.

Attached to this post are crash logs with better redaction / alignment.

Thanks.

My best guess is this is a bug at our end. The crash was triggered by a memory access exception. The crashing thread looks like this:

Thread 20 Crashed:
0 … libdispatch.dylib … dispatch_source_set_timer + 36 …
1 … CFNetwork         … URLConnectionLoader::loadWithWhatToDo(NSURLRequest*, _CFCachedURLResponse const*, long, URLConnectionLoader::WhatToDo) + 1128 …
…
5 … libdispatch.dylib … _dispatch_call_block_and_release + 32 …

Frame 1 is CFNetwork code being called by Dispatch. I took a look at this code and it’s scheduling a timeout timer. Frame 0 shows Dispatch crashing while it tries to work with this timer. This strongly suggests that the timer that CFNetwork passed in is bogus, and so it’s some sort of object lifetime issue within CFNetwork.

Now, it’s always possible that there’s a bug in your app that’s triggering this. For example, you might have a memory management issue that just happens to trigger this problem. However, that seems unlikely. We’ve seen crashes very similar to this being reported in a wide variety of different apps.

Still, if you’re not already doing this routinely, I recommend that you run your app’s test suite with the standard memory debugging tools to see if that flushes out anything actionable.

Normally at this point I’d point you to a bug that Apple is using to track this. However, the bug situation here is kinda muddied, so I’m going to recommend that you file your own bug about this.

Please post your bug number, just for the record.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Is there any news regarding this issue?

What was your bug number?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Gathered another crash log where the main thread state is also captured. It seems to have to do with multithreading + cookie management in CFNetwork. Does this help finding the cause?

Perhaps the issue is in PLCrashReporter, or in the interplay between PLCrashReporter and CFNetwork?

Perhaps.

Frankly, I don’t trust any crash reports other than Apple crash reports. See Implementing Your Own Crash Reporter for an explanation as to why.

Bug number is [FB10663350]

Thanks for that.

Digging into the crash report a little more, I can confirm that the specific crash is caused by CFNetwork calling dispatch_source_set_timer [1] with NULL in the first parameter. Indeed, if you write a tiny test project with this code:

dispatch_source_set_timer(NULL, 0, 0, 0);

frame 0 of the crash will exactly match the crash report [2]:

(lldb) disas -s 0x10274ac3c -c 9
libdispatch.dylib`_dispatch_source_set_runloop_timer_4CF:
    0x10274ac3c <+0>:  pacibsp 
    0x10274ac40 <+4>:  stp    x26, x25, [sp, #-0x50]!
    0x10274ac44 <+8>:  stp    x24, x23, [sp, #0x10]
    0x10274ac48 <+12>: stp    x22, x21, [sp, #0x20]
    0x10274ac4c <+16>: stp    x20, x19, [sp, #0x30]
    0x10274ac50 <+20>: stp    x29, x30, [sp, #0x40]
    0x10274ac54 <+24>: add    x29, sp, #0x40
    0x10274ac58 <+28>: mov    x19, x0
->  0x10274ac5c <+32>: ldr    x25, [x0, #0x58]
(lldb) p/a $x0
(unsigned long) $0 = 0x0000000000000000

This suggests that this isn’t memory corruption, which is more evidence to support my earlier guess is that is a bug in CFNetwork itself.

Debugging this without a reproducible case is going to be hard.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] Not _dispatch_source_set_runloop_timer_4CF! That is a bogus symbolication.

[2] Yeah, and it’s weird that LLDB is also mis-symbolicating this.

This came up in another context so I took a look at the state of the bug. The bug referenced above, FB10663350, was marked as a dup of another bug, (r. 26038149). That, in turn, was marked as fixed in an early iOS 16 release (16.1 I think). If you’re still seeing it on later iOS 16 builds or iOS 17, you should definitely file a new bug about it.

Please post your bug number, just for the record.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Just noticed the in another context was referring to my own post. I've reported a bug and the bug number is FB13447648.

This appears to be an issue on macOS 12.2.1 (21D62) too. Just received some crash logs from a customer that seems like it is the same crash signature (this is one of four different crashes that occured over the course of a week or so. Every crash has the same this basic signature):

Date/Time:             2024-03-25 11:37:15.6659 -0400
OS Version:            macOS 12.2.1 (21D62)
Report Version:        12
Anonymous UUID:        11FA4AA4-0231-994D-2404-EA518B20693B


Time Awake Since Boot: 860000 seconds

System Integrity Protection: enabled

Crashed Thread:        91  Dispatch queue: com.apple.CFNetwork.LoaderQ

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000058
Exception Codes:       0x0000000000000001, 0x0000000000000058
Exception Note:        EXC_CORPSE_NOTIFY

Termination Reason:    Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process:   exc handler [87810]

...

Thread 91 Crashed::  Dispatch queue: com.apple.CFNetwork.LoaderQ
0   libdispatch.dylib             	       0x19ed97ce0 _dispatch_source_set_runloop_timer_4CF + 32
1   CFNetwork                     	       0x1a3d997dc 0x1a3bb1000 + 2000860
2   CFNetwork                     	       0x1a3d997dc 0x1a3bb1000 + 2000860
3   CFNetwork                     	       0x1a3d99e9c 0x1a3bb1000 + 2002588
4   CFNetwork                     	       0x1a3bcba88 0x1a3bb1000 + 109192
5   CFNetwork                     	       0x1a3c4df40 0x1a3bb1000 + 642880
6   libdispatch.dylib             	       0x19ed80e60 _dispatch_call_block_and_release + 32
7   libdispatch.dylib             	       0x19ed82bac _dispatch_client_callout + 20
8   libdispatch.dylib             	       0x19ed8a330 _dispatch_lane_serial_drain + 672
9   libdispatch.dylib             	       0x19ed8aed8 _dispatch_lane_invoke + 444
10  libdispatch.dylib             	       0x19ed8c1c0 _dispatch_workloop_invoke + 1796
11  libdispatch.dylib             	       0x19ed95708 _dispatch_workloop_worker_thread + 656
12  libsystem_pthread.dylib       	       0x19ef3d304 _pthread_wqthread + 288
13  libsystem_pthread.dylib       	       0x19ef3c018 start_wqthread + 8

Thread 91 crashed with ARM Thread State (64-bit):
    x0: 0x0000000000000000   x1: 0x000012ec4f6a5421   x2: 0xffffffffffffffff   x3: 0x000000003b9aca00
    x4: 0x0000000000000000   x5: 0x0000000000000017   x6: 0x0000000000000000   x7: 0x0000000000000000
    x8: 0x000000003b9aca00   x9: 0x800012ec4f6a5421  x10: 0x0000000055d4a800  x11: 0x0000000039387000
   x12: 0x00000000016e3600  x13: 0x000000000003fca8  x14: 0x00000000ee269000  x15: 0x00000001f85f5540
   x16: 0x000000019ed97cc0  x17: 0x00000001f9a44938  x18: 0x0000000000000000  x19: 0x0000000000000000
   x20: 0x000060000064ee60  x21: 0x000060000003d210  x22: 0x0000000000000000  x23: 0x0000000000000000
   x24: 0x0000000000000000  x25: 0x0000000000000000  x26: 0x0000000000000114  x27: 0x000060000187c140
   x28: 0x000060000187c140   fp: 0x000000016d8ee660   lr: 0x15290001a3d997dc
    sp: 0x000000016d8ee620   pc: 0x000000019ed97ce0 cpsr: 0x20001000
   far: 0x0000000000000058  esr: 0x92000006 (Data Abort) byte read Translation fault

It seems to somewhat be related to multiple threads initiating HTTP requests at once, as the crash seems to correlate to a log from our process which prints out some info about a HTTP request getting squashed (ie Mar 25 11:37:15 --- last message repeated 1 time ---)

Unfortunately I haven't found a reliable way to reproduce this yet.

However, after looking into this a bit further (and not knowing which version of libdispatch is included in macOS 12.2.1 (21D62)), from looking at the source for a version of libdispatch, it does seem to point to a null ptr deref is causing the crash (assuming x0: 0x0000000000000000 is getting passed in as first argument):

void
_dispatch_source_set_runloop_timer_4CF(dispatch_source_t ds,
		dispatch_time_t start, uint64_t interval, uint64_t leeway)
{
	// Don't serialize through the source queue for CF timers <rdar://13833190>
	_dispatch_source_set_timer(ds, start, interval, leeway, false);
}

DISPATCH_ALWAYS_INLINE
static inline void
_dispatch_source_set_timer(dispatch_source_t ds, dispatch_time_t start,
		uint64_t interval, uint64_t leeway, bool source_sync)
{
	if (slowpath(!ds->ds_is_timer) ||
			slowpath(ds_timer(ds->ds_refs).flags & DISPATCH_TIMER_INTERVAL)) {
		DISPATCH_CLIENT_CRASH(ds, "Attempt to set timer on a non-timer source");
	}
    ...
}

Id guess the offset of ds_is_timer in the dispatch_source_t struct happens to be 0x58, which is causing the KERN_INVALID_ADDRESS at 0x0000000000000058.

The crash in _dispatch_source_set_runloop_timer_4CF seems back in iOS 18.0 but with a different stack trace:

Launch Time:     2024-11-20T12:12:59Z
OS Version:      iPadOS 18.0 (22A3354)
Report Version:  104

Exception Type:  SIGSEGV
Exception Codes: SEGV_MAPERR at 0x1d
Crashed Thread:  33

Thread 33 Crashed:
0   libdispatch.dylib                    0x00000001a10e2e80 _dispatch_source_set_runloop_timer_4CF + 36
1   CFNetwork                            0x000000019a8e801c URLConnectionLoader::loadWithWhatToDo(NSURLRequest*, _CFCachedURLResponse const*, long, URLConnectionLoader::WhatToDo) + 972
2   CFNetwork                            0x000000019a975920 ___ZN19URLConnectionLoader29continueWithCacheLookupResultEP12NSURLRequestPK20_CFCachedURLResponse23NSURLRequestCachePolicy_block_invoke + 96
3   CFNetwork                            0x000000019a96f880 ___ZNK19URLConnectionLoader25withExistingProtocolAsyncEU13block_pointerFvP11URLProtocolE_block_invoke + 36
4   CFNetwork                            0x000000019a96f30c ___ZNK18QCoreSchedulingSet12performAsyncEU13block_pointerFvvE_block_invoke + 56
5   libdispatch.dylib                    0x00000001a10cc370 _dispatch_call_block_and_release + 28
6   libdispatch.dylib                    0x00000001a10ce0d0 _dispatch_client_callout + 16
7   libdispatch.dylib                    0x00000001a10d56d8 _dispatch_lane_serial_drain + 740
8   libdispatch.dylib                    0x00000001a10d6214 _dispatch_lane_invoke + 428
9   libdispatch.dylib                    0x00000001a10d7670 _dispatch_workloop_invoke + 1728
10  libdispatch.dylib                    0x00000001a10e1258 _dispatch_root_queue_drain_deferred_wlh + 284
11  libdispatch.dylib                    0x00000001a10e0aa4 _dispatch_workloop_worker_thread + 536
12  libsystem_pthread.dylib              0x00000002206cbc7c _pthread_wqthread + 284
13  libsystem_pthread.dylib              0x00000002206c8488 start_wqthread + 4