Change in iOS 14 Beta 3 to trigger 0xdead10cc

FYI, TN2151 has effectively been obsoleted by Diagnosing Issues Using Crash Reports and Device Logs and related articles. Specifically, the official doc for exception codes like this Understanding the Exception Types in a Crash Report.

A problem like this could be a system bug or it could be a benign change in system behaviour that exposes a bug on your side. It’s kinda hard to say with more details. However, IMO it’s always worth filing a bug for stuff like this, in that the system is supposed to be binary compatible from release to release.

Simultaneously, I think it would behoove you to investigate exactly which file is triggering this problem. One trick is to log all the files you have open at the time you become eligible for suspension and then, when you crash, look through that list for likely suspects.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

One trick is to log all the files you have open

The mechanism for doing this is not exactly obvious, so I’ve pasted in a code snippet below.

IMPORTANT This is for debugging purposes only. Using code like this in production raises all sorts of issues, most critically that other threads can be changing the file descriptor table while you’re walking it.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

---

Code Block  
func openFilePaths() -> [String] {
		(0..<getdtablesize()).map { fd in
				// Return "" for invalid file descriptors.
				var flags: CInt = 0
				guard fcntl(fd, F_GETFL, &flags) >= 0 else {
						return ""
				}
				// Return "?" for file descriptors not associated with a path, for
				// example, a socket.
				var path = [CChar](repeating: 0, count: Int(MAXPATHLEN))
				guard fcntl(fd, F_GETPATH, &path) >= 0 else {
						return "?"
				}
				return String(cString: path)
		}
}

Thanks Quinn, I stumbled upon some C code for that and was writing it up in Swift too, thanks for saving me the time and helping me debug the cause.

I was able to narrow down the file that is triggering the problem as I was able to reproduce this on my personal device so I could see the relevant Console log regarding suspension and termination:

Code Block json
[application<…>:3879] Terminating with context: <RBSTerminateContext| domain:15 code:0xDEAD10CC explanation:[application<…>:3879] was suspended with locked system files:
/var/mobile/Containers/Shared/AppGroup/A66EB78A-2BBC-49D4-BDEA-6A2AF7E8A5A6/default.realm.lock
not in allowed directories:
/var/mobile/Containers/Data/Application/E1435A44-ABC6-4254-B547-B5423D9FCAB1
/var/mobile/Containers/Data/Application/E1435A44-ABC6-4254-B547-B5423D9FCAB1/tmp reportType:CrashLog maxTerminationResistance:Interactive>

This points to Realm's default.realm.lock being the locked file which is not permitted as it sits within the App Group container as opposed to the app's own container (which I presume is the first 'allowed directory').

Whilst this explains the cause of the crash on beta 3 it doesn't explain why this only started occurring on beta 3.

I will file a bug report as you suggest with respect to binary compatibility but any insights you might be able to provide now we've narrowed down the affected file would be much appreciated too!

I have filed bug report FB8128103. I wasn't 100% on which component to select so chose UIKit, however if this seems better placed set to something else on your side please do tweak/update so it lands with the right team.

This does appear to be a b3-unique thing, and judging by the crash logs I'm seeing on-device, it's affecting system apps like Spotlight and Safari as well. I also am unable to reproduce it at all when building using Xcode 12 to device (only Xcode 11-built binaries seem to reproduce it), but that may be a coincidence.

I have also filed FB8116961 for this one.

In my own experiences with two projects, either building from Xcode 11.6 or Xcode 12 beta 3, the issue of Realm lock file in shared group did exist on real devices. If you use the simulator, everything is fine.

Realm 5.3.2.

There is an issue on realm too. But it seams people believe it is Apple who to blame.

https://github.com/realm/realm-cocoa/issues/6671

Just to add to what's already been said here: This issue is re-producible when running an affected app on an actual iOS device running iOS 14 Beta 3 (not on the simulator), even with the debugger attached, as follows :-

1. Build & Run an affected app on an actual iOS device running iOS 14 Beta 3.

2. Swipe up from the bottom (or press the home button if your device has one) while the app is in the foreground to send it to the background.

That's it: A short moment later, the system kills the app, and Xcode's Console reads:

Code Block 
Message from debugger: Terminated due to signal 9

I have also filed FB8214214 for this issue (and included a quick & dirty sample project).

Just updated to beta 4 and the issue persists.

For those who had filed issues to Apple. Did you get any replies? As this issue still exists in beta 4 now.

Nothing yet for my bug report, still marked as Open.

Is there any updates about this issue? Or any way to walkaround this issue? Apps keep crashing...

iOS 14 beta 4 still happens "Termination Reason: Namespace RUNNINGBOARD, Code 0xdead10cc"

The issue persists in beta 5.

Hey guys, I just reproduce the bug without Realm. It's all about holding flock in App Group directory. You can follow these steps to quickly reproduce the weird crash:

1. Create a new iOS App project (Objective-C) using Xcode.

2. Add an App Group container in project setting page.

3. Paste this code snippet in application:didFinishLaunchingWithOptions::

Code Block objc
    // 1. prepare a non-empty file under App Group directory
    NSFileManager* fileManager = [NSFileManager defaultManager];
    NSURL* dir = [fileManager containerURLForSecurityApplicationGroupIdentifier:@"group.***.***....."]; <= your group id here
    NSURL* fileUrl = [dir URLByAppendingPathComponent:@"file"];
    [fileManager createFileAtPath:[fileUrl path]
                         contents:[@"some data..." dataUsingEncoding:(NSUTF8StringEncoding)]
                       attributes:nil];
    
    // 2. hold a file lock
    int fd = open([fileUrl path].UTF8String, O_RDWR);
    int ret = flock(fd, LOCK_SH);

4. Debug the project on a physical device running iOS 14 b3/b4/b5.
5. The app will be killed after you return to home screen.
6. If you unlock the file by calling flock(fd, LOCK_UN) before the app enters suspended state, the app won't be killed by iOS.

---

Note that:

• This only crash on a physical device, not a simulator.

• Xcode does not handle it like a normal crash. It just print a termination message in console and ends the debug session gracefully.

If you unlock the file by calling flock(fd, LOCK_UN) before the app
enters suspended state, the app won't be killed by iOS.

Right, this is expected. Holdling file locks while your app is suspended is exactly what triggers the 0xdead10cc termination. You must release file locks before coming eligible for suspension.

The issue here is that previously working code is now being hit by this check. I took a look at our investigation of this (r. 66931425) but it’s too early for me to post any concrete details. I hope that’ll change soon (-:

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

The issue here is that previously working code is now being hit by this check. I took a look at our investigation of this (r. 66931425) but it’s too early for me to post any concrete details. I hope that’ll change soon (-:

Unfortunately the response on my bug report FB8128103 doesn't seem to suggest any investigation and despite it being a binary compatibility issue the response I received, in the form of the following message, asked me to close my report:

This is an issue specific to a third-party, not an Apple issue.

This is a Realm bug that they’re tracking:

https://github.com/realm/realm-cocoa/issues/6671

Please contact Realm for further support.

Please close your feedback report, or let us know if this is still an issue for you.

I've responded accordingly re. binary compatibility though so fingers crossed this gets another look.

I'm trying to reproduce this issue with beta 6, but haven't been able to do it. Can anyone confirm that they still see it? I have an extension and an app both using flock to take a lock on a file in a shared container and the app appears to be able to suspend just fine while holding the lock.

This is using flock(fileHandle.fileDescriptor, LOCK_EX | LOCK_NB).

The crash still happened on my app with beta 6. I'm waiting for the realm core release with the fix for this issue.

I'm trying to reproduce this issue with beta 6, but haven't been able to do it. Can anyone confirm that they still see it? I have an extension and an app both using flock to take a lock on a file in a shared container and the app appears to be able to suspend just fine while holding the lock.

This is using 

Code Block 
flock(fileHandle.fileDescriptor, LOCK_EX | LOCK_NB)

.

I can confirm that this is still an issue on beta 6. To reproduce, use the following to get a file descriptor, instead of NSFileHandle.

Code Block 
int fd = open([fileUrl path].UTF8String, O_RDWR);

and then, the following to acquire a file lock:-

Code Block 
int ret = flock(fd, LOCK_SH);

The crash still happened on my app with beta 6.

Right, that’s in line with my expectations. I’m still hoping to see change here, but it’s not in iOS 14.0b6.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

I can confirm that this is still an issue on beta 6. To reproduce, use the following to get a file descriptor, instead of NSFileHandle.

int fd = open([fileUrl path].UTF8String, ORDWR);

and then, the following to acquire a file lock:-

int ret = flock(fd, LOCKSH);

The FileHandle was just to release the fd when the object went away. This code right here never crashes for me:

Code Block swift
    let lock2URL = FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: sharedContainerIdentifier)!.appendingPathComponent("shared2.lock")
    var fd: Int32 = -1
    try? Data().write(to: lock2URL, options: .atomic)
    fd = open(lock2URL.path, O_RDWR)
    assert(fd >= 0)
    assert(flock(fd, LOCK_SH) == 0)

followed by backgrounding the app and verifying that it is suspended in the debugger.

I'm testing on an iPhone 6s with iOS 14 beta 6.

This appears to have been resolved as of beta 7.

Now that this issue has been resolved, I want to explain a bit of the backstory.

It’s always unsafe for an app to hold a file lock while it’s suspended. If some other app [1] attempts to take that lock, it’ll end up waiting indefinitely. This is an annoying source of app hangs. Moreover, the user can’t clear the hang by terminating the just-launched app. They must either terminate the suspended app or restart the device.

A while back Apple enhanced the watchdog to detect this and terminate the offending app with the 0xdead10cc exception code. However, systems prior to iOS 14 had a bug such that this check did not function properly in all cases. Thus, it was possible for an app to suspend while holding a lock and not be terminated with 0xdead10cc.

In iOS 14 we fixed that bug. iOS is now able to detect this problem correctly in all cases.

Unfortunately this has uncovered a number of file locking bugs in third-party apps. An app that previously got away with being suspended while holding a file lock was now being terminated with 0xdead10cc. This change in behaviour caused a certain amount of consternation, as indicated by this thread.

To avoid this we’ve changed iOS 14.0b7 and later to be more forgiving in this case (r. 66931425). I should stress that this only prevents the app from being terminated, it does not fix the underlying problem. The ‘rogue’ file lock could still cause a hang in some other app.

---

Finally, some concrete advice:

• Avoid file locks where possible.

• For this sort of task, consider using Core Data, which is built in to the system and thus will do the right thing.

• If you must use a file lock, make sure to release it before you become eligible for suspension. For example, you might wrap this work in a UIApplication background task that prevents the app from being suspended while the work is in progress.

• If you’re using a third-party library, check that it follows these rules. If it doesn’t, raise this issue with the library’s vendor.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"

[1] In this context I’m using app to refer to either an app or an app extension.

Finally, some concrete advice:
Avoid file locks where possible.
For this sort of task, consider using Core Data, which is built in to the system and thus will do the right thing.

My app and widget extension use Core Data to access the same persistent store via a shared app group and are frequently getting this crash. Any advice Eskimo?

Code Block 
Exception Type:  EXC_CRASH (SIGKILL)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note:  EXC_CORPSE_NOTIFY
Termination Reason: Namespace RUNNINGBOARD, Code 0xdead10cc
Triggered by Thread:  0
Thread 0 name:
Thread 0 Crashed:
0   libsystem_kernel.dylib        	0x00000001c5b5fdd0 mach_msg_trap + 8
1   libsystem_kernel.dylib        	0x00000001c5b5f184 mach_msg + 76 (mach_msg.c:103)
2   CoreFoundation                	0x0000000199bcecf8 CFRunLoopServiceMachPort + 380 (CFRunLoop.c:2641)
3   CoreFoundation                	0x0000000199bc8ea8 CFRunLoopRun + 1216 (CFRunLoop.c:2974)
4   CoreFoundation                	0x0000000199bc84bc CFRunLoopRunSpecific + 600 (CFRunLoop.c:3242)
5   Foundation                    	0x000000019ae45e30 -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 232 (NSRunLoop.m:374)
6   Foundation                    	0x000000019ae7879c -[NSRunLoop(NSRunLoop) run] + 92 (NSRunLoop.m:399)
7   libxpc.dylib                  	0x00000001e11da428 _xpc_objc_main + 688 (main.m:265)
8   libxpc.dylib                  	0x00000001e11dc700 xpc_main + 180 (init.c:1198)
9   Foundation                    	0x000000019ae7aaa8 -[NSXPCListener resume] + 316 (NSXPCListener.m:448)
10  PlugInKit                     	0x00000001c830b564 -[PKService run] + 424 (PKService.m:186)
11  PlugInKit                     	0x00000001c830b1a4 +[PKService main] + 576 (PKService.m:115)
12  PlugInKit                     	0x00000001c830b970 +[PKService _defaultRun:arguments:] + 24 (PKService.m:233)
13  ExtensionKit                  	0x000000019e0b9848 EXExtensionMain + 84 (EXExtensionMain.m:23)
14  Foundation                    	0x000000019afc2b78 NSExtensionMain + 200 (NSExtensionMain.m:13)
15  libdyld.dylib                 	0x000000019988fe60 start + 4
Thread 2:
0   libsystem_kernel.dylib        	0x00000001c5b84f6c fsync + 8
1   libsqlite3.dylib              	0x00000001b2ab496c unixSync + 212 (sqlite3.c:39882)
2   libsqlite3.dylib              	0x00000001b2abe3dc syncJournal + 488 (sqlite3.c:23756)
3   libsqlite3.dylib              	0x00000001b2ab43cc sqlite3PagerCommitPhaseOne + 1216 (sqlite3.c:65303)
4   libsqlite3.dylib              	0x00000001b2a9f164 sqlite3BtreeCommitPhaseOne + 164 (sqlite3.c:75145)
5   libsqlite3.dylib              	0x00000001b2a69f60 sqlite3VdbeHalt + 2820 (sqlite3.c:87147)
6   libsqlite3.dylib              	0x00000001b2a97cdc sqlite3VdbeExec + 56848 (sqlite3.c:95898)
7   libsqlite3.dylib              	0x00000001b2a887ac sqlite3_step + 304 (sqlite3.c:90618)
8   CoreData                      	0x000000019fd9c12c _execute + 116 (NSSQLiteConnection.m:4480)
9   CoreData                      	0x000000019fbddd94 -[NSSQLiteConnection commitTransaction] + 296 (NSSQLiteConnection.m:3184)
10  CoreData                      	0x000000019fbecc58 -[NSSQLiteConnection createSchema] + 548 (NSSQLiteConnection.m:7371)
11  CoreData                      	0x000000019fbe8190 -[NSSQLiteConnection connect] + 1940 (NSSQLiteConnection.m:2280)
12  CoreData                      	0x000000019fd75044 32-[NSSQLCore _loadAndSetMetadata]_block_invoke + 132 (NSSQLCore.m:1840)
13  CoreData                      	0x000000019fd99d3c 37-[NSSQLiteConnection performAndWait:]_block_invoke + 48 (NSSQLiteConnection.m:718)
14  libdispatch.dylib             	0x0000000199850ac8 _dispatch_client_callout + 20 (object.m:559)
15  libdispatch.dylib             	0x000000019985ec8c _dispatch_lane_barrier_sync_invoke_and_complete + 60 (queue.c:998)
16  CoreData                      	0x000000019fbe36fc -[NSSQLiteConnection performAndWait:] + 172 (NSSQLiteConnection.m:715)
17  CoreData                      	0x000000019fbe8764 -[NSSQLCore _loadAndSetMetadata] + 340 (NSSQLCore.m:1865)
18  CoreData                      	0x000000019fbe894c -[NSSQLCore loadMetadata:] + 32 (NSSQLCore.m:1886)
19  CoreData                      	0x000000019fd4a8ec __91-[NSPersistentStoreCoordinator addPersistentStoreWithType:configuration:URL:options:error:]_block_invoke + 1508 (NSPersistentStoreCoordinator.m:1673)
20  CoreData                      	0x000000019fd57f08 gutsOfBlockToNSPersistentStoreCoordinatorPerform + 208 (NSPersistentStoreCoordinator.m:361)
21  libdispatch.dylib             	0x0000000199850ac8 _dispatch_client_callout + 20 (object.m:559)
22  libdispatch.dylib             	0x000000019985ec8c _dispatch_lane_barrier_sync_invoke_and_complete + 60 (queue.c:998)
23  CoreData                      	0x000000019fd47cbc _perform + 184 (NSPersistentStoreCoordinator.m:483)
24  CoreData                      	0x000000019fbe1024 -[NSPersistentStoreCoordinator addPersistentStoreWithType:configuration:URL:options:error:] + 484 (NSPersistentStoreCoordinator.m:1530)
25  CoreData                      	0x000000019fd49d98 -[NSPersistentStoreCoordinator _doAddPersistentStoreWithDescription:privateCopy:completeOnMainThread:withHandler:] + 544 (NSPersistentStoreCoordinator.m:1431)
26  libdispatch.dylib             	0x000000019984efd0 _dispatch_call_block_and_release + 32 (init.c:1454)
27  libdispatch.dylib             	0x0000000199850ac8 _dispatch_client_callout + 20 (object.m:559)
28  libdispatch.dylib             	0x0000000199853a10 _dispatch_queue_override_invoke + 692 (inline_internal.h:2548)
29  libdispatch.dylib             	0x0000000199861104 _dispatch_root_queue_drain + 356 (inline_internal.h:2589)
30  libdispatch.dylib             	0x00000001998618e8 _dispatch_worker_thread2 + 116 (queue.c:6766)
31  libsystem_pthread.dylib       	0x00000001e11ac8cc _pthread_wqthread + 216 (pthread.c:2210)
32  libsystem_pthread.dylib       	0x00000001e11b377c start_wqthread + 8

My app and widget extension use Core Data to access the same
persistent store via a shared app group and are frequently getting
this crash.

Can you post a full crash report? Use the text attachment feature so it doesn’t clutter up the timeline.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@apple.com"