Overview
I am at a total loss here on where to start diagnosing this issue. We have built an app using Flutter, it is live with ~15k users, and due to the nature of things, I am unable to post direct source publicly (more than happy to share snippets, or discuss in a more private setting).
Essentially, we released a feature, and immediately began getting a handful of reports saying that users' iOS phones were restarting automatically within a minute or so of using this new screen. I'm fairly confident this is iOS 13+.
I have been able to replicate this issue only twice on our iPhone SE; never on our iPhone X. It took 20minutes of me being idle on this particular screen for it to happen (despite several reports that it is happening in far less time). Since then, I have rigorously run the app through profiling tools: there are no memory leaks, no alarming CPU usage, and the energy profiler reads "low" with only GPU/Network spikes when expected.
Logs
There are no direct logs or crashes relating to our app specifically. The best I have found is the original restart log from the device:
Incident Identifier: A4F0C8D3-3359-4BDD-A8C9-AD4A4A0709CA CrashReporter Key: 6bb3f35ef170b67fd0897ab58b6394977d126dec Date: 2019-11-20 08:57:01.07 -0600 Reset count: 1 Boot failure count: 0 Boot faults: Boot stage: 0 Boot app: 2681261667I used the CrashReporter Key to identify the following:
{ "build" : "iPhone OS 13.2.2 (17B102)", "product" : "iPhone8,4", "kernel" : "Darwin Kernel Version 19.0.0: Wed Oct 9 22:41:51 PDT 2019; root:xnu-6153.42.1~1\/RELEASE_ARM64_S8000", "tuning" : { }, "incident" : "4643EE6D-EE53-44A6-B53B-9C96553AD250", "crashReporterKey" : "6bb3f35ef170b67fd0897ab58b6394977d126dec", "date" : "2019-11-16 10:05:08.90 -0600", "reason" : "Power assertion timeout for \"RTLocationAwarenessManager\". Simply releasing it now.\nBacktrace for power assertion: pid 31 (0x1a1781fa4 0x1c419a80c 0x1c42a3058 0x1c42a32b8 0x19f25e610 0x19f25f184 0x19f20b404 0x19f20be28 0x19f215314 0x19f2aef88 0x19f2b1ad4)", "frontmostPids" : [ 54 ], "exception" : "0xdeaffeed", "absoluteTime" : 47310107548, "memoryStatus" :Which led me to believe it may be power related (hence my profiling and checking energy usage).
But I also found:
{ "build" : "iPhone OS 13.2.2 (17B102)", "product" : "iPhone8,4", "kernel" : "Darwin Kernel Version 19.0.0: Wed Oct 9 22:41:51 PDT 2019; root:xnu-6153.42.1~1\/RELEASE_ARM64_S8000", "incident" : "0723F54F-C499-4E02-A271-8029D025E81A", "crashReporterKey" : "6bb3f35ef170b67fd0897ab58b6394977d126dec", "date" : "2019-11-20 09:20:09.66 -0600", "panicString" : "panic(cpu 1 caller 0xfffffff0133eb460): mbuf_watchdog: 4 waiters stuck for 14 secs\n39939\/39953 mbufs in use:\n\t39927 mbufs allocated to data\n\t12 mbufs allocated to packet headers\n\t14 mbufs allocated to caches\n21504\/21504 mbuf 2KB clusters in use\n0\/168 mbuf 4KB clusters in use\n4\/28 mbuf 16KB clusters in use\n54451 KB allocated to network (approx. 97% in use)\n2 KB returned to the system\nVM allocation failures: contiguous 0, normal 0, one page 0\nworker thread runs: 889, expansions: 27, cl 248\/0, bigcl 0\/0, 16k 640\/5245\nworker thread last run time: 1372 (14 seconds ago)\ndrain routine last run time: 1313 (73 seconds ago)\n\nmbuf leak detection table:\n\ttotal captured: 1478 (one per 500)\n\ttotal allocs outstanding: 45\n\tnew hash recorded: 1384 allocs, 1318 traces\n\thash collisions: 29 allocs, 4 traces\n\toverwrites: 908 allocs, 0 traces\n\tlock conflicts: 0\n\ntop 5 outstanding traces:\n[1] 36 outstanding alloc(s), 39 hit(s), 1 collision(s)\n[2] 6 outstanding alloc(s), 24 hit(s), 0 collision(s)\n[3] 1 outstanding alloc(s), 40 hit(s), 0 collision(s)\n[4] 1 outstanding alloc(s), 1 hit(s), 0 collision(s)\n[5] 1 outstanding alloc(s), 12 hit(s), 3 collision(s)\n\n trace [1] trace [2] trace [3] \ttrace [4] trace [5] \n ------------------ ------------------ ------------------ ------------------ ------------------ \n 1: 0xfffffff00750fd2c 0xfffffff00750fd2c 0xfffffff00750fd2c 0xfffffff00750fd2c 0xfffffff00750fd2c \n 2: 0xfffffff0074ecb88 0xfffffff0074ecb88 0xfffffff0074ecb88 0xfffffff0074ecb88 0xfffffff0074ecbc4 \n 3: 0xfffffff00750e904 0xfffffff00751fe10 0xfffffff0075131d4 0xfffffff007393dfc 0xfffffff007514c68 \n 4: 0xfffffff0074ecb60 0xfffffff0074f71e0 0xfffffff00751fed0 0xfffffff0073aac1c 0xfffffff00751fdb0 \n 5: 0xfffffff007514c68 0xfffffff0074f081c 0xfffffff0074f71e0 0xfffffff0075085ec 0xfffffff00752e604 \n 6: 0xfffffff00751fdb0 0xfffffff0074f05e8 0xfffffff0074f081c 0xfffffff007509840 0xfffffff00752e40c \n 7: 0xfffffff0074f71e0 0xfffffff0075af628 0xfffffff0074f05e8 0xfffffff00748af98 0xfffffff0075af628 \n 8: 0xfffffff0074f081c 0xfffffff0071f2ef4 0xfffffff0075af628 0xfffffff0071169f0 0xfffffff0071f2ef4 \n 9: 0xfffffff0074f05e8 0xfffffff0070bd5c4 0xfffffff0071f2ef4 0xfffffff0070bd5c4 \n10: 0xfffffff0075af628 0xfffffff0070bd5c4 \n11: 0xfffffff0071f2ef4 \n12: 0xfffffff0070bd5c4 \n13: \n14: \n15: \n16: \n\nDebugger message: panic\nMemory ID: 0x6\nOS version: 17B102\nKernel version: Darwin Kernel Version 19.0.0: Wed Oct 9 22:41:51 PDT 2019; root:xnu-6153.42.1~1\/RELEASE_ARM64_S8000\nKernelCache UUID: E5568684C25D539C58C559CD9B2AA4B6\nKernel UUID: C0C04BDB-6A5F-3FE0-AD0E-29F05BB687DB\niBoot version: iBoot-5540.40.51\nsecure boot?: YES\nPaniclog version: 13\nKernel slide: 0x000000000bedc000\nKernel text base: 0xfffffff012ee0000\nmach_absolute_time: 0x7bf4b1bd3\nEpoch Time: sec usec\n Boot : 0x5dd55438 0x000be275\n Sleep : 0x00000000 0x00000000\n Wake : 0x00000000 0x00000000\n Calendar: 0x5dd5599c 0x000c4327\n\nPanicked task 0xffffffe001c4fc00: 1451 pages, 5 threads: pid 36: atc\nPanicked thread: 0xffffffe004836f88, backtrace: 0xffffffe00014b070, tid: 9069\n\t\t lr: 0xfffffff012fd0c90 fp: 0xffffffe00014b0b0\n\t\t lr: 0xfffffff012fd0af0 fp: 0xffffffe00014b120\n\t\t lr: 0xfffffff0130cee28 fp: 0xffffffe00014b1c0\n\t\t lr: 0xfffffff012f995c4 fp: 0xffffffe00014b1d0\n\t\t lr: 0xfffffff012fd0468 fp: 0xffffffe00014b540\n\t\t lr: 0xfffffff012fd07b0 fp: 0xffffffe00014b590\n\t\t lr: 0xfffffff01357be28 fp: 0xffffffe00014b5b0\n\t\t lr: 0xfffffff0133eb460 fp: 0xffffffe00014b650\n\t\t lr: 0xfffffff0133c8b60 fp: 0xffffffe00014b6f0\n\t\t lr: 0xfffffff0133ea904 fp: 0xffffffe00014b7a0\n\t\t lr: 0xfffffff0133c8b60 fp: 0xffffffe00014b840\n\t\t lr: 0xfffffff0133f0c68 fp: 0xffffffe00014b8e0\n\t\t lr: 0xfffffff0133fbdb0 fp: 0xffffffe00014ba00\n\t\t lr: 0xfffffff0133d31e0 fp: 0xffffffe00014ba30\n\t\t lr: 0xfffffff0133cc81c fp: 0xffffffe00014bae0\n\t\t lr: 0xfffffff0133cc5e8 fp: 0xffffffe00014bb50\n\t\t lr: 0xfffffff01348b628 fp: 0xffffffe00014bbe0\n\t\t lr: 0xfffffff0130ceef4 fp: 0xffffffe00014bc80\n\t\t lr: 0xfffffff012f995c4 fp: 0xffffffe00014bc90\n\t\t lr: 0x00000001910a0b9c fp: 0x0000000000000000\n\n", "panicFlags" : "0x2", "otherString" : "\n** Stackshot Succeeded ** Bytes Traced 350736 **\n", "memoryStatus" : {"compressorSize":1109,"compressions":8209,"decompressions":3042,"busyBufferCount":0,"pageSize":16384,"memoryPressure":false,"memoryPages":{"active":51206,"throttled":0,"fileBacked":49710,"wired":17735,"purgeable":1858,"inactive":50873,"free":1426,"speculative":343}}, "processByPid" : { "0" : {"timesThrottled":0,"turnstileInfo":["thread 1796: blocked on 575, hops: 1, priority: 92","thread 8917: blocked on 575, hops: 1, priority: 92","thread 8934: blocked on 546, hops: 1, priority: 92","thread 8968: blocked on 546, hops: 1, priority: 92","thread 8975: blocked on 546, hops: 1, priority: 92","thread 8976: blocked on 546, hops: 1, priority: 92","thread 8982: blocked on 546, hops: 1, priority: 92"],"pageIns":0,"waitInfo":["thread 1796: kernel mutex 0x5c0cfa02fd9af5e3 owned by thread 575","thread 3080: semaphore port 0 with unknown owner","thread 4381: semaphore port 0 with unknown owner","thread 4382: semaphore port 0 with unknown owner","thread 4383: semaphore port 0 with unknown owner","thread 8917: kernel mutex 0x5c0cfa02fd9af5e3 owned by thread 575","thread 8934: kernel mutex 0x5c0cfa02fd8b1563 owned by thread 546","thread 8968: kernel mutex 0x5c0cfa02fd8b1563 owned by thread 546","thread 8975: kernel mutex 0x5c0cfa02fd8b1563 owned by thread 546","thread 8976: kernel mutex 0x5c0cfa02fd8b1563 owned by thread 546","thread 8982: kernel mutex 0x5c0cfa02fd8b1563 owned by thread 546"],"timesDidThrottle":0,"procname":"kernel_task","copyOnWriteFaults":0,"threadById":{"606":{"continuation":[0,68839822204],"userTime":5.4160000000000003e-06,"systemTime":0,"id":606,"basePriority":81,"name":"dlil_input_pdp_ip1","user_usec":5,"schedPriority":81,"system_usec":0,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241958928611]},"445":{"continuation":[0,68842226148],"userTime":0.011681208,"systemTime":0,"id":445,"basePriority":81,"user_usec":11681,"system_usec":0,"schedPriority":81,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[0,68844608688]},"259":{"continuation":[0,68843352704],"userTime":4.2910000000000001e-06,"systemTime":0,"id":259,"basePriority":81,"name":"AppleT700XTempSensor","user_usec":4,"schedPriority":81,"system_usec":0,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241954305003]},"348":{"continuation":[0,68843352704],"userTime":5.0000000000000004e-06,"systemTime":0,"id":348,"basePriority":81,"name":"AppleS5L8960XDART","user_usec":5,"schedPriority":81,"system_usec":0,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241954146603]},"534":{"continuation":[0,68843352704],"userTime":1.8330000000000001e-06,"systemTime":0,"id":534,"basePriority":81,"name":"IOAsynchronousScheduler","user_usec":1,"schedPriority":81,"system_usec":0,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241953966219]},"551":{"continuation":[0,68843352704],"userTime":1.916e-06,"systemTime":0,"id":551,"basePriority":81,"name":"AppleCS42L71Audio","user_usec":1,"schedPriority":81,"system_usec":0,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241953963979]},"615":{"continuation":[0,68839849260],"userTime":2.2910000000000002e-06,"systemTime":0,"id":615,"basePriority":81,"name":"ifnet_start_pdp_ip3","user_usec":2,"schedPriority":81,"system_usec":0,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241958942763]},"454":{"continuation":[0,68841994852],"userTime":1.2500000000000001e-06,"systemTime":0,"id":454,"basePriority":81,"user_usec":1,"system_usec":0,"schedPriority":81,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[0,68844611616]},"543":{"continuation":[0,68843352704],"userTime":1.666e-06,"systemTime":0,"id":543,"basePriority":81,"user_usec":1,"system_usec":0,"schedPriority":81,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241953968219]},"607":{"continuation":[0,68839849260],"userTime":2.9160000000000001e-06,"systemTime":0,"id":607,"basePriority":81,"name":"ifnet_start_pdp_ip1","user_usec":2,"schedPriority":81,"system_usec":0,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241958926323]},"357":{"userTime":0.025310915999999999,"systemTime":0,"name":"RTBuddy","id":357,"basePriority":81,"user_usec":25310,"system_usec":0,"schedPriority":81,"kernelFrames":[[0,68838034924],[0,68838029360],[0,68843352756],[0,68837717140]],"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241954147323]},"446":{"continuation":[0,68841720552],"userTime":1.9999999999999999e-06,"systemTime":0,"id":446,"basePriority":81,"user_usec":2,"system_usec":0,"schedPriority":81,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[0,68844597624]},"349":{"continuation":[0,68843352704],"userTime":4.5000000000000001e-06,"systemTime":0,"id":349,"basePriority":81,"name":"AppleS5L8960XDART","user_usec":4,"schedPriority":81,"system_usec":0,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241954219499]},"438":{"continuation":[0,68843352704],"userTime":9.7916000000000002e-05,"systemTime":0,"id":438,"basePriority":81,"name":"AppleARMPMUCharger","user_usec":97,"schedPriority":81,"system_usec":0,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241954141163]},"285":{"continuation":[0,68843352704],"userTime":5.541e-06,"systemTime":0,"id":285,"basePriority":81,"name":"AppleVTempDispatcher","user_usec":5,"schedPriority":81,"system_usec":0,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241954303163]},"293":{"continuation":[0,68843352704],"userTime":5.75e-06,"systemTime":0,"id":293,"basePriority":81,"name":"AppleSSE","user_usec":5,"schedPriority":81,"system_usecThis is seemingly a serious issue, and the reports we have are very consistent with how to reproduce. Unfortunately, I came to Flutter with a native Android background, so I am not heavily familiar with iOS and how to properly track down something like this.
Has anyone seen similar behavior before, and where would I start for diagnosing something like this?
