1 Reply
      Latest reply on Nov 21, 2019 1:13 AM by eskimo
       calebisstupid Level 1 Level 1 (0 points)

        Overview

         

        I am at a total loss here on where to start diagnosing this issue. We have built an app using Flutter, it is live with ~15k users, and due to the nature of things, I am unable to post direct source publicly (more than happy to share snippets, or discuss in a more private setting).

         

        Essentially, we released a feature, and immediately began getting a handful of reports saying that users' iOS phones were restarting automatically within a minute or so of using this new screen. I'm fairly confident this is iOS 13+.

         

        I have been able to replicate this issue only twice on our iPhone SE; never on our iPhone X. It took 20minutes of me being idle on this particular screen for it to happen (despite several reports that it is happening in far less time). Since then, I have rigorously run the app through profiling tools: there are no memory leaks, no alarming CPU usage, and the energy profiler reads "low" with only GPU/Network spikes when expected.

         

        Logs

        There are no direct logs or crashes relating to our app specifically. The best I have found is the original restart log from the device:

        Incident Identifier: A4F0C8D3-3359-4BDD-A8C9-AD4A4A0709CA CrashReporter Key:   6bb3f35ef170b67fd0897ab58b6394977d126dec Date: 2019-11-20 08:57:01.07 -0600 Reset count: 1 Boot failure count: 0 Boot faults: Boot stage: 0 Boot app: 2681261667 

        I used the CrashReporter Key to identify the following:

        { "build" : "iPhone OS 13.2.2 (17B102)", "product" : "iPhone8,4", "kernel" : "Darwin Kernel Version 19.0.0: Wed Oct  9 22:41:51 PDT 2019; root:xnu-6153.42.1~1\/RELEASE_ARM64_S8000", "tuning" : {  }, "incident" : "4643EE6D-EE53-44A6-B53B-9C96553AD250", "crashReporterKey" : "6bb3f35ef170b67fd0897ab58b6394977d126dec", "date" : "2019-11-16 10:05:08.90 -0600", "reason" : "Power assertion timeout for \"RTLocationAwarenessManager\".  Simply releasing it now.\nBacktrace for power assertion: pid 31 (0x1a1781fa4 0x1c419a80c 0x1c42a3058 0x1c42a32b8 0x19f25e610 0x19f25f184 0x19f20b404 0x19f20be28 0x19f215314 0x19f2aef88 0x19f2b1ad4)", "frontmostPids" : [ 54 ], "exception" : "0xdeaffeed", "absoluteTime" : 47310107548, "memoryStatus" : 

        Which led me to believe it may be power related (hence my profiling and checking energy usage).
        But I also found:

        { "build" : "iPhone OS 13.2.2 (17B102)", "product" : "iPhone8,4", "kernel" : "Darwin Kernel Version 19.0.0: Wed Oct  9 22:41:51 PDT 2019; root:xnu-6153.42.1~1\/RELEASE_ARM64_S8000", "incident" : "0723F54F-C499-4E02-A271-8029D025E81A", "crashReporterKey" : "6bb3f35ef170b67fd0897ab58b6394977d126dec", "date" : "2019-11-20 09:20:09.66 -0600", "panicString" : "panic(cpu 1 caller 0xfffffff0133eb460): mbuf_watchdog: 4 waiters stuck for 14 secs\n39939\/39953 mbufs in use:\n\t39927 mbufs allocated to data\n\t12 mbufs allocated to packet headers\n\t14 mbufs allocated to caches\n21504\/21504 mbuf 2KB clusters in use\n0\/168 mbuf 4KB clusters in use\n4\/28 mbuf 16KB clusters in use\n54451 KB allocated to network (approx. 97% in use)\n2 KB returned to the system\nVM allocation failures: contiguous 0, normal 0, one page 0\nworker thread runs: 889, expansions: 27, cl 248\/0, bigcl 0\/0, 16k 640\/5245\nworker thread last run time: 1372 (14 seconds ago)\ndrain routine last run time: 1313 (73 seconds ago)\n\nmbuf leak detection table:\n\ttotal captured: 1478 (one per 500)\n\ttotal allocs outstanding: 45\n\tnew hash recorded: 1384 allocs, 1318 traces\n\thash collisions: 29 allocs, 4 traces\n\toverwrites: 908 allocs, 0 traces\n\tlock conflicts: 0\n\ntop 5 outstanding traces:\n[1] 36 outstanding alloc(s), 39 hit(s), 1 collision(s)\n[2] 6 outstanding alloc(s), 24 hit(s), 0 collision(s)\n[3] 1 outstanding alloc(s), 40 hit(s), 0 collision(s)\n[4] 1 outstanding alloc(s), 1 hit(s), 0 collision(s)\n[5] 1 outstanding alloc(s), 12 hit(s), 3 collision(s)\n\n    trace [1]           trace [2]           trace [3]       \ttrace [4]           trace [5]      \n    ------------------  ------------------  ------------------      ------------------  ------------------ \n 1: 0xfffffff00750fd2c  0xfffffff00750fd2c  0xfffffff00750fd2c  0xfffffff00750fd2c  0xfffffff00750fd2c  \n 2: 0xfffffff0074ecb88  0xfffffff0074ecb88  0xfffffff0074ecb88  0xfffffff0074ecb88  0xfffffff0074ecbc4  \n 3: 0xfffffff00750e904  0xfffffff00751fe10  0xfffffff0075131d4  0xfffffff007393dfc  0xfffffff007514c68  \n 4: 0xfffffff0074ecb60  0xfffffff0074f71e0  0xfffffff00751fed0  0xfffffff0073aac1c  0xfffffff00751fdb0  \n 5: 0xfffffff007514c68  0xfffffff0074f081c  0xfffffff0074f71e0  0xfffffff0075085ec  0xfffffff00752e604  \n 6: 0xfffffff00751fdb0  0xfffffff0074f05e8  0xfffffff0074f081c  0xfffffff007509840  0xfffffff00752e40c  \n 7: 0xfffffff0074f71e0  0xfffffff0075af628  0xfffffff0074f05e8  0xfffffff00748af98  0xfffffff0075af628  \n 8: 0xfffffff0074f081c  0xfffffff0071f2ef4  0xfffffff0075af628  0xfffffff0071169f0  0xfffffff0071f2ef4  \n 9: 0xfffffff0074f05e8  0xfffffff0070bd5c4  0xfffffff0071f2ef4                      0xfffffff0070bd5c4  \n10: 0xfffffff0075af628                      0xfffffff0070bd5c4                                          \n11: 0xfffffff0071f2ef4                                                                                  \n12: 0xfffffff0070bd5c4                                                                                  \n13:                                                                                                     \n14:                                                                                                     \n15:                                                                                                     \n16:                                                                                                     \n\nDebugger message: panic\nMemory ID: 0x6\nOS version: 17B102\nKernel version: Darwin Kernel Version 19.0.0: Wed Oct  9 22:41:51 PDT 2019; root:xnu-6153.42.1~1\/RELEASE_ARM64_S8000\nKernelCache UUID: E5568684C25D539C58C559CD9B2AA4B6\nKernel UUID: C0C04BDB-6A5F-3FE0-AD0E-29F05BB687DB\niBoot version: iBoot-5540.40.51\nsecure boot?: YES\nPaniclog version: 13\nKernel slide:     0x000000000bedc000\nKernel text base: 0xfffffff012ee0000\nmach_absolute_time: 0x7bf4b1bd3\nEpoch Time:        sec       usec\n  Boot    : 0x5dd55438 0x000be275\n  Sleep   : 0x00000000 0x00000000\n  Wake    : 0x00000000 0x00000000\n  Calendar: 0x5dd5599c 0x000c4327\n\nPanicked task 0xffffffe001c4fc00: 1451 pages, 5 threads: pid 36: atc\nPanicked thread: 0xffffffe004836f88, backtrace: 0xffffffe00014b070, tid: 9069\n\t\t  lr: 0xfffffff012fd0c90  fp: 0xffffffe00014b0b0\n\t\t  lr: 0xfffffff012fd0af0  fp: 0xffffffe00014b120\n\t\t  lr: 0xfffffff0130cee28  fp: 0xffffffe00014b1c0\n\t\t  lr: 0xfffffff012f995c4  fp: 0xffffffe00014b1d0\n\t\t  lr: 0xfffffff012fd0468  fp: 0xffffffe00014b540\n\t\t  lr: 0xfffffff012fd07b0  fp: 0xffffffe00014b590\n\t\t  lr: 0xfffffff01357be28  fp: 0xffffffe00014b5b0\n\t\t  lr: 0xfffffff0133eb460  fp: 0xffffffe00014b650\n\t\t  lr: 0xfffffff0133c8b60  fp: 0xffffffe00014b6f0\n\t\t  lr: 0xfffffff0133ea904  fp: 0xffffffe00014b7a0\n\t\t  lr: 0xfffffff0133c8b60  fp: 0xffffffe00014b840\n\t\t  lr: 0xfffffff0133f0c68  fp: 0xffffffe00014b8e0\n\t\t  lr: 0xfffffff0133fbdb0  fp: 0xffffffe00014ba00\n\t\t  lr: 0xfffffff0133d31e0  fp: 0xffffffe00014ba30\n\t\t  lr: 0xfffffff0133cc81c  fp: 0xffffffe00014bae0\n\t\t  lr: 0xfffffff0133cc5e8  fp: 0xffffffe00014bb50\n\t\t  lr: 0xfffffff01348b628  fp: 0xffffffe00014bbe0\n\t\t  lr: 0xfffffff0130ceef4  fp: 0xffffffe00014bc80\n\t\t  lr: 0xfffffff012f995c4  fp: 0xffffffe00014bc90\n\t\t  lr: 0x00000001910a0b9c  fp: 0x0000000000000000\n\n", "panicFlags" : "0x2", "otherString" : "\n** Stackshot Succeeded ** Bytes Traced 350736 **\n", "memoryStatus" : {"compressorSize":1109,"compressions":8209,"decompressions":3042,"busyBufferCount":0,"pageSize":16384,"memoryPressure":false,"memoryPages":{"active":51206,"throttled":0,"fileBacked":49710,"wired":17735,"purgeable":1858,"inactive":50873,"free":1426,"speculative":343}}, "processByPid" : { "0" : {"timesThrottled":0,"turnstileInfo":["thread 1796: blocked on 575, hops: 1, priority: 92","thread 8917: blocked on 575, hops: 1, priority: 92","thread 8934: blocked on 546, hops: 1, priority: 92","thread 8968: blocked on 546, hops: 1, priority: 92","thread 8975: blocked on 546, hops: 1, priority: 92","thread 8976: blocked on 546, hops: 1, priority: 92","thread 8982: blocked on 546, hops: 1, priority: 92"],"pageIns":0,"waitInfo":["thread 1796: kernel mutex 0x5c0cfa02fd9af5e3 owned by thread 575","thread 3080: semaphore port 0 with unknown owner","thread 4381: semaphore port 0 with unknown owner","thread 4382: semaphore port 0 with unknown owner","thread 4383: semaphore port 0 with unknown owner","thread 8917: kernel mutex 0x5c0cfa02fd9af5e3 owned by thread 575","thread 8934: kernel mutex 0x5c0cfa02fd8b1563 owned by thread 546","thread 8968: kernel mutex 0x5c0cfa02fd8b1563 owned by thread 546","thread 8975: kernel mutex 0x5c0cfa02fd8b1563 owned by thread 546","thread 8976: kernel mutex 0x5c0cfa02fd8b1563 owned by thread 546","thread 8982: kernel mutex 0x5c0cfa02fd8b1563 owned by thread 546"],"timesDidThrottle":0,"procname":"kernel_task","copyOnWriteFaults":0,"threadById":{"606":{"continuation":[0,68839822204],"userTime":5.4160000000000003e-06,"systemTime":0,"id":606,"basePriority":81,"name":"dlil_input_pdp_ip1","user_usec":5,"schedPriority":81,"system_usec":0,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241958928611]},"445":{"continuation":[0,68842226148],"userTime":0.011681208,"systemTime":0,"id":445,"basePriority":81,"user_usec":11681,"system_usec":0,"schedPriority":81,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[0,68844608688]},"259":{"continuation":[0,68843352704],"userTime":4.2910000000000001e-06,"systemTime":0,"id":259,"basePriority":81,"name":"AppleT700XTempSensor","user_usec":4,"schedPriority":81,"system_usec":0,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241954305003]},"348":{"continuation":[0,68843352704],"userTime":5.0000000000000004e-06,"systemTime":0,"id":348,"basePriority":81,"name":"AppleS5L8960XDART","user_usec":5,"schedPriority":81,"system_usec":0,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241954146603]},"534":{"continuation":[0,68843352704],"userTime":1.8330000000000001e-06,"systemTime":0,"id":534,"basePriority":81,"name":"IOAsynchronousScheduler","user_usec":1,"schedPriority":81,"system_usec":0,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241953966219]},"551":{"continuation":[0,68843352704],"userTime":1.916e-06,"systemTime":0,"id":551,"basePriority":81,"name":"AppleCS42L71Audio","user_usec":1,"schedPriority":81,"system_usec":0,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241953963979]},"615":{"continuation":[0,68839849260],"userTime":2.2910000000000002e-06,"systemTime":0,"id":615,"basePriority":81,"name":"ifnet_start_pdp_ip3","user_usec":2,"schedPriority":81,"system_usec":0,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241958942763]},"454":{"continuation":[0,68841994852],"userTime":1.2500000000000001e-06,"systemTime":0,"id":454,"basePriority":81,"user_usec":1,"system_usec":0,"schedPriority":81,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[0,68844611616]},"543":{"continuation":[0,68843352704],"userTime":1.666e-06,"systemTime":0,"id":543,"basePriority":81,"user_usec":1,"system_usec":0,"schedPriority":81,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241953968219]},"607":{"continuation":[0,68839849260],"userTime":2.9160000000000001e-06,"systemTime":0,"id":607,"basePriority":81,"name":"ifnet_start_pdp_ip1","user_usec":2,"schedPriority":81,"system_usec":0,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241958926323]},"357":{"userTime":0.025310915999999999,"systemTime":0,"name":"RTBuddy","id":357,"basePriority":81,"user_usec":25310,"system_usec":0,"schedPriority":81,"kernelFrames":[[0,68838034924],[0,68838029360],[0,68843352756],[0,68837717140]],"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241954147323]},"446":{"continuation":[0,68841720552],"userTime":1.9999999999999999e-06,"systemTime":0,"id":446,"basePriority":81,"user_usec":2,"system_usec":0,"schedPriority":81,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[0,68844597624]},"349":{"continuation":[0,68843352704],"userTime":4.5000000000000001e-06,"systemTime":0,"id":349,"basePriority":81,"name":"AppleS5L8960XDART","user_usec":4,"schedPriority":81,"system_usec":0,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241954219499]},"438":{"continuation":[0,68843352704],"userTime":9.7916000000000002e-05,"systemTime":0,"id":438,"basePriority":81,"name":"AppleARMPMUCharger","user_usec":97,"schedPriority":81,"system_usec":0,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241954141163]},"285":{"continuation":[0,68843352704],"userTime":5.541e-06,"systemTime":0,"id":285,"basePriority":81,"name":"AppleVTempDispatcher","user_usec":5,"schedPriority":81,"system_usec":0,"state":["TH_WAIT","TH_UNINT"],"waitEvent":[1,6632951241954303163]},"293":{"continuation":[0,68843352704],"userTime":5.75e-06,"systemTime":0,"id":293,"basePriority":81,"name":"AppleSSE","user_usec":5,"schedPriority":81,"system_usec 

        This is seemingly a serious issue, and the reports we have are very consistent with how to reproduce. Unfortunately, I came to Flutter with a native Android background, so I am not heavily familiar with iOS and how to properly track down something like this.

        Has anyone seen similar behavior before, and where would I start for diagnosing something like this?

        • Re: Kernel Panic on select iOS 13 devices
          eskimo Apple Staff Apple Staff (12,435 points)

          You are correct that the iPhone spontaneously rebooting is a strong indicator that you hit a kernel panic.  Third-party apps should never be able to induce a kernel panic; such panics are automatically bugworthy.

          Most kernel panics triggered by user space code are related to graphics drivers.  However, the last log you posted suggest a networking problem (note the message mbuf_watchdog: 4 waiters stuck for 14 secs).

          Just to confirm, does the timestamp on that log corelate with the time the device rebooted?

          Share and Enjoy

          Quinn “The Eskimo!”
          Apple Developer Relations, Developer Technical Support, Core OS/Hardware
          let myEmail = "eskimo" + "1" + "@apple.com"