Change in iOS 14 Beta 3 to trigger 0xdead10cc

We've noticed that our app is now immediately crashing on background on iOS 14 Beta 3 with the 0xdead10cc termination reason. This indicates (as per Technical Note TN2151) that the app held on to a file lock or sqlite database lock during suspension.

However this crash didn't occur at all prior to beta 3.

Did something change in iOS that might cause this? Stricter enforcement?

This is a hard crash for us to debug with little than the technical note to go on and potentially an unknown change in iOS itself.

Replies

The issue here is that previously working code is now being hit by this check. I took a look at our investigation of this (r. 66931425) but it’s too early for me to post any concrete details. I hope that’ll change soon (-:

Unfortunately the response on my bug report FB8128103 doesn't seem to suggest any investigation and despite it being a binary compatibility issue the response I received, in the form of the following message, asked me to close my report:

This is an issue specific to a third-party, not an Apple issue.

This is a Realm bug that they’re tracking:

https://github.com/realm/realm-cocoa/issues/6671

Please contact Realm for further support.

Please close your feedback report, or let us know if this is still an issue for you.

I've responded accordingly re. binary compatibility though so fingers crossed this gets another look.

I'm trying to reproduce this issue with beta 6, but haven't been able to do it. Can anyone confirm that they still see it? I have an extension and an app both using flock to take a lock on a file in a shared container and the app appears to be able to suspend just fine while holding the lock.

This is using flock(fileHandle.fileDescriptor, LOCK_EX | LOCK_NB).
The crash still happened on my app with beta 6. I'm waiting for the realm core release with the fix for this issue.

I'm trying to reproduce this issue with beta 6, but haven't been able to do it. Can anyone confirm that they still see it? I have an extension and an app both using flock to take a lock on a file in a shared container and the app appears to be able to suspend just fine while holding the lock.

This is using 

Code Block
flock(fileHandle.fileDescriptor, LOCK_EX | LOCK_NB)
.
I can confirm that this is still an issue on beta 6. To reproduce, use the following to get a file descriptor, instead of NSFileHandle.

Code Block
int fd = open([fileUrl path].UTF8String, O_RDWR);


and then, the following to acquire a file lock:-

Code Block
int ret = flock(fd, LOCK_SH);

The crash still happened on my app with beta 6.

Right, that’s in line with my expectations. I’m still hoping to see change here, but it’s not in iOS 14.0b6.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"


I can confirm that this is still an issue on beta 6. To reproduce, use the following to get a file descriptor, instead of NSFileHandle.

int fd = open([fileUrl path].UTF8String, ORDWR);

and then, the following to acquire a file lock:-

int ret = flock(fd, LOCK
SH);

The FileHandle was just to release the fd when the object went away. This code right here never crashes for me:

Code Block swift
    let lock2URL = FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: sharedContainerIdentifier)!.appendingPathComponent("shared2.lock")
    var fd: Int32 = -1
    try? Data().write(to: lock2URL, options: .atomic)
    fd = open(lock2URL.path, O_RDWR)
    assert(fd >= 0)
    assert(flock(fd, LOCK_SH) == 0)


followed by backgrounding the app and verifying that it is suspended in the debugger.

I'm testing on an iPhone 6s with iOS 14 beta 6.
This appears to have been resolved as of beta 7.
Now that this issue has been resolved, I want to explain a bit of the backstory.

It’s always unsafe for an app to hold a file lock while it’s suspended. If some other app [1] attempts to take that lock, it’ll end up waiting indefinitely. This is an annoying source of app hangs. Moreover, the user can’t clear the hang by terminating the just-launched app. They must either terminate the suspended app or restart the device.

A while back Apple enhanced the watchdog to detect this and terminate the offending app with the 0xdead10cc exception code. However, systems prior to iOS 14 had a bug such that this check did not function properly in all cases. Thus, it was possible for an app to suspend while holding a lock and not be terminated with 0xdead10cc.

In iOS 14 we fixed that bug. iOS is now able to detect this problem correctly in all cases.

Unfortunately this has uncovered a number of file locking bugs in third-party apps. An app that previously got away with being suspended while holding a file lock was now being terminated with 0xdead10cc. This change in behaviour caused a certain amount of consternation, as indicated by this thread.

To avoid this we’ve changed iOS 14.0b7 and later to be more forgiving in this case (r. 66931425). I should stress that this only prevents the app from being terminated, it does not fix the underlying problem. The ‘rogue’ file lock could still cause a hang in some other app.



Finally, some concrete advice:
  • Avoid file locks where possible.

  • For this sort of task, consider using Core Data, which is built in to the system and thus will do the right thing.

  • If you must use a file lock, make sure to release it before you become eligible for suspension. For example, you might wrap this work in a UIApplication background task that prevents the app from being suspended while the work is in progress.

  • If you’re using a third-party library, check that it follows these rules. If it doesn’t, raise this issue with the library’s vendor.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

[1] In this context I’m using app to refer to either an app or an app extension.

Finally, some concrete advice:
Avoid file locks where possible.
For this sort of task, consider using Core Data, which is built in to the system and thus will do the right thing.

My app and widget extension use Core Data to access the same persistent store via a shared app group and are frequently getting this crash. Any advice Eskimo?

Code Block
Exception Type: EXC_CRASH (SIGKILL)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY
Termination Reason: Namespace RUNNINGBOARD, Code 0xdead10cc
Triggered by Thread: 0
Thread 0 name:
Thread 0 Crashed:
0 libsystem_kernel.dylib 0x00000001c5b5fdd0 mach_msg_trap + 8
1 libsystem_kernel.dylib 0x00000001c5b5f184 mach_msg + 76 (mach_msg.c:103)
2 CoreFoundation 0x0000000199bcecf8 CFRunLoopServiceMachPort + 380 (CFRunLoop.c:2641)
3 CoreFoundation 0x0000000199bc8ea8 CFRunLoopRun + 1216 (CFRunLoop.c:2974)
4 CoreFoundation 0x0000000199bc84bc CFRunLoopRunSpecific + 600 (CFRunLoop.c:3242)
5 Foundation 0x000000019ae45e30 -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 232 (NSRunLoop.m:374)
6 Foundation 0x000000019ae7879c -[NSRunLoop(NSRunLoop) run] + 92 (NSRunLoop.m:399)
7 libxpc.dylib 0x00000001e11da428 _xpc_objc_main + 688 (main.m:265)
8 libxpc.dylib 0x00000001e11dc700 xpc_main + 180 (init.c:1198)
9 Foundation 0x000000019ae7aaa8 -[NSXPCListener resume] + 316 (NSXPCListener.m:448)
10 PlugInKit 0x00000001c830b564 -[PKService run] + 424 (PKService.m:186)
11 PlugInKit 0x00000001c830b1a4 +[PKService main] + 576 (PKService.m:115)
12 PlugInKit 0x00000001c830b970 +[PKService _defaultRun:arguments:] + 24 (PKService.m:233)
13 ExtensionKit 0x000000019e0b9848 EXExtensionMain + 84 (EXExtensionMain.m:23)
14 Foundation 0x000000019afc2b78 NSExtensionMain + 200 (NSExtensionMain.m:13)
15 libdyld.dylib 0x000000019988fe60 start + 4
Thread 2:
0 libsystem_kernel.dylib 0x00000001c5b84f6c fsync + 8
1 libsqlite3.dylib 0x00000001b2ab496c unixSync + 212 (sqlite3.c:39882)
2 libsqlite3.dylib 0x00000001b2abe3dc syncJournal + 488 (sqlite3.c:23756)
3 libsqlite3.dylib 0x00000001b2ab43cc sqlite3PagerCommitPhaseOne + 1216 (sqlite3.c:65303)
4 libsqlite3.dylib 0x00000001b2a9f164 sqlite3BtreeCommitPhaseOne + 164 (sqlite3.c:75145)
5 libsqlite3.dylib 0x00000001b2a69f60 sqlite3VdbeHalt + 2820 (sqlite3.c:87147)
6 libsqlite3.dylib 0x00000001b2a97cdc sqlite3VdbeExec + 56848 (sqlite3.c:95898)
7 libsqlite3.dylib 0x00000001b2a887ac sqlite3_step + 304 (sqlite3.c:90618)
8 CoreData 0x000000019fd9c12c _execute + 116 (NSSQLiteConnection.m:4480)
9 CoreData 0x000000019fbddd94 -[NSSQLiteConnection commitTransaction] + 296 (NSSQLiteConnection.m:3184)
10 CoreData 0x000000019fbecc58 -[NSSQLiteConnection createSchema] + 548 (NSSQLiteConnection.m:7371)
11 CoreData 0x000000019fbe8190 -[NSSQLiteConnection connect] + 1940 (NSSQLiteConnection.m:2280)
12 CoreData 0x000000019fd75044 32-[NSSQLCore _loadAndSetMetadata]_block_invoke + 132 (NSSQLCore.m:1840)
13 CoreData 0x000000019fd99d3c 37-[NSSQLiteConnection performAndWait:]_block_invoke + 48 (NSSQLiteConnection.m:718)
14 libdispatch.dylib 0x0000000199850ac8 _dispatch_client_callout + 20 (object.m:559)
15 libdispatch.dylib 0x000000019985ec8c _dispatch_lane_barrier_sync_invoke_and_complete + 60 (queue.c:998)
16 CoreData 0x000000019fbe36fc -[NSSQLiteConnection performAndWait:] + 172 (NSSQLiteConnection.m:715)
17 CoreData 0x000000019fbe8764 -[NSSQLCore _loadAndSetMetadata] + 340 (NSSQLCore.m:1865)
18 CoreData 0x000000019fbe894c -[NSSQLCore loadMetadata:] + 32 (NSSQLCore.m:1886)
19 CoreData 0x000000019fd4a8ec __91-[NSPersistentStoreCoordinator addPersistentStoreWithType:configuration:URL:options:error:]_block_invoke + 1508 (NSPersistentStoreCoordinator.m:1673)
20 CoreData 0x000000019fd57f08 gutsOfBlockToNSPersistentStoreCoordinatorPerform + 208 (NSPersistentStoreCoordinator.m:361)
21 libdispatch.dylib 0x0000000199850ac8 _dispatch_client_callout + 20 (object.m:559)
22 libdispatch.dylib 0x000000019985ec8c _dispatch_lane_barrier_sync_invoke_and_complete + 60 (queue.c:998)
23 CoreData 0x000000019fd47cbc _perform + 184 (NSPersistentStoreCoordinator.m:483)
24 CoreData 0x000000019fbe1024 -[NSPersistentStoreCoordinator addPersistentStoreWithType:configuration:URL:options:error:] + 484 (NSPersistentStoreCoordinator.m:1530)
25 CoreData 0x000000019fd49d98 -[NSPersistentStoreCoordinator _doAddPersistentStoreWithDescription:privateCopy:completeOnMainThread:withHandler:] + 544 (NSPersistentStoreCoordinator.m:1431)
26 libdispatch.dylib 0x000000019984efd0 _dispatch_call_block_and_release + 32 (init.c:1454)
27 libdispatch.dylib 0x0000000199850ac8 _dispatch_client_callout + 20 (object.m:559)
28 libdispatch.dylib 0x0000000199853a10 _dispatch_queue_override_invoke + 692 (inline_internal.h:2548)
29 libdispatch.dylib 0x0000000199861104 _dispatch_root_queue_drain + 356 (inline_internal.h:2589)
30 libdispatch.dylib 0x00000001998618e8 _dispatch_worker_thread2 + 116 (queue.c:6766)
31 libsystem_pthread.dylib 0x00000001e11ac8cc _pthread_wqthread + 216 (pthread.c:2210)
32 libsystem_pthread.dylib 0x00000001e11b377c start_wqthread + 8

My app and widget extension use Core Data to access the same
persistent store via a shared app group and are frequently getting
this crash.

Can you post a full crash report? Use the text attachment feature so it doesn’t clutter up the timeline.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"



My app and widget extension use Core Data to access the same
persistent store via a shared app group and are frequently getting
this crash.
Can you post a full crash report? Use the text attachment feature so it doesn’t clutter up the timeline.
Attached an example of the crash. Thanks for taking a look.
Thanks for the crash report. Unfortunately I don’t have any good insight into this. My advice is that you start a new thread for this issue, making sure to:
  • Include your crash report (again as a text attachment), or possibly just reference back to this thread.

  • Tag it with Core Data, so that folks with more experience with Core Data can find it.

Hopefully someone will respond there but, if not, open a DTS tech support incident so that you can discuss this with our Core Data specialist.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

To avoid this we’ve changed iOS 14.0b7 and later to be more forgiving in this case (r. 66931425).

If you must use a file lock, make sure to release it before you become eligible for suspension.

I wonder how "future-proof" this forgiving treatment is.

Thing is, I'm using file locks in a bit of inconventional way: to find out which processes in the same app group are actually alive. It works like this:

  • When a process (a main app, share extension, etc.) starts, it obtains a unique "virtual process ID" from a shared SQLite DB, creates a file with that ID in a shared app container and puts a file lock on it.
  • This virtual PID is used a key/token to put a temporary modification protection on shared virtual resources (basically, files in a shared VFS).
  • When other process wants such resource and "sees" that it is protected by some other VPID, it can query VPID DB and/or relevant VPID file's lock to see if such process still actually exist and still wants the resource for itself. If process (e.g. a share extension) have finished normally, it will simply delete it's VPID file (raising a file lock along the way) and DB record; if it was abruptly terminated by the OS, the DB record will stay, but the file lock will be raised automagically.

As you can see, this use case would not produce any deadlocks (since lock on a each file is put exactly once and only queried afterwards), but totally relies on ability to hold file locks in background. So, I wonder how valid/future-proof is that.