On iOS 10.x devices, TestFlight app runs but release app crashes (same exact build)

Can you share a crash report? See this link for where to find them.

Here is a crash report for the release version (exact same build as the TestFlight beta, which doesn't crash). The release version crashes immediately when the app is run. I don't think it loads any of my code at all.

This crash is reproducible. If you use an iOS 10 device to run the release version of my app (details in bug report FB7692530), it will crash. If you run the TestFlight beta, it will not crash.

Crash report follows:

Incident Identifier: 28D0FA6B-4AB1-4C37-A0A7-EF284E419E4C

CrashReporter Key: 3fcddbe8a8f49dd3eca438c0983d3a49990d48e5

Hardware Model: iPhone5,2

Process: <appname> [328]

Path: /private/var/containers/Bundle/Application/55AC7D1A-83D5-44B7-B55D-F1061318BBA3/<appname>.app/<appname>

Identifier: com.<company>.<appname>

Version: <version> (<build>)

Code Type: ARM (Native)

Role: Foreground

Parent Process: launchd [1]

Coalition: com.<company>.<appname> [402]

Date/Time: 2020-05-07 13:32:39.7419 +0100

Launch Time: 2020-05-07 13:32:37.0000 +0100

OS Version: iPhone OS 10.3.4 (14G61)

Report Version: 104

Exception Type: EXC_BAD_ACCESS (SIGSEGV)

Exception Subtype: KERN_INVALID_ADDRESS at 0x0a8a00d4

Termination Signal: Segmentation fault: 11

Termination Reason: Namespace SIGNAL, Code 0xb

Terminating Process: exc handler [0]

Triggered by Thread: 0

Thread 0 name:

Thread 0 Crashed:

0 ??? 0x0a8a00d4 0 + 176816340

Thread 1:

0 libsystem_pthread.dylib 0x1af9a454 start_wqthread + 0

Thread 2:

0 libsystem_kernel.dylib 0x1aee673c __workq_kernreturn + 8

1 libsystem_pthread.dylib 0x1af9a8ea _pthread_wqthread + 1150 (pthread.c:2205)

2 libsystem_pthread.dylib 0x1af9a45c start_wqthread + 8

Thread 3:

0 libsystem_kernel.dylib 0x1aee673c __workq_kernreturn + 8

1 libsystem_pthread.dylib 0x1af9a8ea _pthread_wqthread + 1150 (pthread.c:2205)

2 libsystem_pthread.dylib 0x1af9a45c start_wqthread + 8

Thread 4:

0 libsystem_kernel.dylib 0x1aee673c __workq_kernreturn + 8

1 libsystem_pthread.dylib 0x1af9a8ea _pthread_wqthread + 1150 (pthread.c:2205)

2 libsystem_pthread.dylib 0x1af9a45c start_wqthread + 8

Thread 5 name:

Thread 5:

0 libsystem_kernel.dylib 0x1aed0900 mach_msg_trap + 20

1 libsystem_kernel.dylib 0x1aed06e0 mach_msg + 44 (mach_msg.c:103)

2 CoreFoundation 0x1b6d3be2 __CFRunLoopServiceMachPort + 144 (CFRunLoop.c:2527)

3 CoreFoundation 0x1b6d2064 __CFRunLoopRun + 1436 (CFRunLoop.c:2870)

4 CoreFoundation 0x1b6251ae CFRunLoopRunSpecific + 470 (CFRunLoop.c:3113)

5 CoreFoundation 0x1b624fd0 CFRunLoopRunInMode + 104 (CFRunLoop.c:3143)

6 Foundation 0x1bf79af4 -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 258 (NSRunLoop.m:367)

7 Foundation 0x1bf9676c -[NSRunLoop(NSRunLoop) runUntilDate:] + 86 (NSRunLoop.m:411)

8 UIKit 0x212aead8 -[UIEventFetcher threadMain] + 128 (UIEventFetcher.m:313)

9 Foundation 0x1c05d8ea __NSThread__start__ + 1122 (NSThread.m:1163)

10 libsystem_pthread.dylib 0x1af9c93a _pthread_body + 216 (pthread.c:697)

11 libsystem_pthread.dylib 0x1af9c85c _pthread_start + 234 (pthread.c:744)

12 libsystem_pthread.dylib 0x1af9a468 thread_start + 8

Thread 6:

0 libsystem_pthread.dylib 0x1af9a454 start_wqthread + 0

Thread 7 name:

Thread 7:

0 libsystem_kernel.dylib 0x1aed0900 mach_msg_trap + 20

1 libsystem_kernel.dylib 0x1aed06e0 mach_msg + 44 (mach_msg.c:103)

2 CoreFoundation 0x1b6d3be2 __CFRunLoopServiceMachPort + 144 (CFRunLoop.c:2527)

3 CoreFoundation 0x1b6d2064 __CFRunLoopRun + 1436 (CFRunLoop.c:2870)

4 CoreFoundation 0x1b6251ae CFRunLoopRunSpecific + 470 (CFRunLoop.c:3113)

5 CoreFoundation 0x1b624fd0 CFRunLoopRunInMode + 104 (CFRunLoop.c:3143)

6 AVFAudio 0x3223682e GenericRunLoopThread::Entry(void*) + 142 (GenericRunLoopThread.h:106)

7 AVFAudio 0x3225f58e CAPThread::Entry(CAPThread*) + 154

8 libsystem_pthread.dylib 0x1af9c93a _pthread_body + 216 (pthread.c:697)

9 libsystem_pthread.dylib 0x1af9c85c _pthread_start + 234 (pthread.c:744)

10 libsystem_pthread.dylib 0x1af9a468 thread_start + 8

Thread 8 name:

Thread 8:

0 libsystem_kernel.dylib 0x1aed0900 mach_msg_trap + 20

1 libsystem_kernel.dylib 0x1aed06e0 mach_msg + 44 (mach_msg.c:103)

2 CoreFoundation 0x1b6d3be2 __CFRunLoopServiceMachPort + 144 (CFRunLoop.c:2527)

3 CoreFoundation 0x1b6d2064 __CFRunLoopRun + 1436 (CFRunLoop.c:2870)

4 CoreFoundation 0x1b6251ae CFRunLoopRunSpecific + 470 (CFRunLoop.c:3113)

5 CoreFoundation 0x1b624fd0 CFRunLoopRunInMode + 104 (CFRunLoop.c:3143)

6 AudioToolbox 0x1e2441b6 GenericRunLoopThread::Entry(void*) + 142 (GenericRunLoopThread.h:106)

7 AudioToolbox 0x1e425fce CAPThread::Entry(CAPThread*) + 154

8 libsystem_pthread.dylib 0x1af9c93a _pthread_body + 216 (pthread.c:697)

9 libsystem_pthread.dylib 0x1af9c85c _pthread_start + 234 (pthread.c:744)

10 libsystem_pthread.dylib 0x1af9a468 thread_start + 8

Thread 9:

0 libsystem_pthread.dylib 0x1af9a454 start_wqthread + 0

Thread 10:

0 libsystem_pthread.dylib 0x1af9a454 start_wqthread + 0

Thread 0 crashed with ARM Thread State (32-bit):

r0: 0xffffffff r1: 0x0a8a00d5 r2: 0x00000030 r3: 0x00000078

r4: 0x43870000 r5: 0x00000000 r6: 0x00000000 r7: 0x00000000

r8: 0x00000000 r9: 0x0037b838 r10: 0x0037ba08 r11: 0x165b9b50

ip: 0x0037b92c sp: 0x0037b850 lr: 0x001238f1 pc: 0x0a8a00d4

cpsr: 0x60000030

[binary images cut] - If you need this, please let me know.

The smashed stack in Thread 0 means this could be related to the issue you called out from the Xcode 11.5 beta release notes. However, the sp register looks like an ok at first glance -- some of the other crashes due to the bug you mentioned have bad values in that register.

You may not have reproduced this crash while in TestFlight becuase you didn't hit the exact right code path. If you look at the lr register, it should be within the range for your app specified in the Binary Images. If it's in that range, you can use the atos command line tool to symbolicate that memory address and figure out what function your code expected to return to had this crash not occured. Instructions on atos are at the end of this article.

Based on this part of the binary images list:

Binary Images:

0x95000 - 0x16cfff <appname> armv7 <a0e4299e63db3fbebeb8bf66d17ec612> /var/containers/Bundle/Application/55AC7D1A-83D5-44B7-B55D-F1061318BBA3/<appname>.app/<appname>

I ran the atos command to look up the address at the link register and got this result:

% atos -arch armv7 -o <appname>.app.dSYM/Contents/Resources/DWARF/<appname> -l 0x95000 0x001238f1

EVP_CipherInit_ex (in <appname>) + 69

This is a function in the openssl library which I built as a static library and linked into my app. I'm using the same library that I built in 2015, and it has been working on iOS 10 devices until now.

The only part of the app that uses the openssl library is the code that parses the in-app purchase receipt.

Doesn't TestFlight also generate its own in-app purchase receipt if the tester buys an in-app purchase? If so, then both the TestFlight and release versions should run the same openssl code.

I figured out the issue.

First, I reproduced the crash on TestFlight by making an in-app purchase, which created a receipt and triggered the openssl code. Thanks for pointing me to the link register, which helped me figure out how to reproduce the crash.

Then, I found the problem was caused by selecting the "Include bitcode for iOS content" option when submitting the app to TestFlight/App Store. My theory is that the openssl library (which I built with embedded bitcode) is being recompiled by Apple, using a buggy compiler (possibly the one in Xcode 11.4, which is known to generate incorrect code for armv7).

When I unchecked the "Include bitcode for iOS content" option, I was able to run the TestFlight app and make an in-app purchase without the app crashing.

I had this confirmed by Apple DTS for an exactly similar crash problem: the App Store uses the buggy compiler, mangles the app so that 32-bit devices can't run it, and then goes ahead and installs it anyway.

Our crash also involved OpenSSL, and the recommended cure is indeed to turn off "Include bitcode" because then the mangling doesn't happen. In fact you can reproduce the problem for yourself locally by taking a suitable build, archiving it, pressing "Distribute", taking the "Development" route, and turning ON the "Rebuild from Bitcode" option.

Xcode 11.5 cures this problem locally, so the problem will be cured when the App Store publication process starts using tools from that version. When I asked, the App Store people said that the version of the tools they use was not something they are prepared to reveal, so we will never know when it is safe to turn on "Include bitcode" again.

Thanks for confirming this exact situation and also for the tip on how to reproduce locally without TestFlight. Since Apple DTS knew about this after your ticket, I wish they had reverted their compiler to a working one!