Kernel panic on kextunload >10.13.5

Hi Team,


We are facing kernel panic while doing kextunload to our driver. This is working fine for the OSX <= 10.13.5.

On OSX >10.13.5 wile unloading the kext, its getting panic.


We are communication with the kernel through iokit userclient , we have agent on the userspace runnung. Without killing the agent while we are trying to do a kextunload to our driver its getting panic. But if we kill our agent then do a kext unload its working fine.


Plese look at the below stack strace of the panic



Invalid connection: com.apple.coresymbolicationd

Anonymous UUID: C7B5663B-2CE3-9CC5-CA18-AD438D317BCD


Tue Aug 14 15:18:50 2018


*** Panic Report ***

panic(cpu 0 caller 0xffffff801ec6e2e1): "com_verdasys_dgagent[0x100000513]::terminate(kIOServiceSynchronous) timeout\n"@/BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-4570.71.2/iokit/Kernel/IOService.cpp:2297

Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64

Kernel slide: 0x000000001e400000

Kernel text base: 0xffffff801e600000

Dgagent text base: 0xffffff7fa0d12000

BSD process name corresponding to current thread: kextunload


0xffffff801e66c1c6 : 0006c1c6 handle_debugger_trap (in kernel) + 1222

0xffffff801e795274 : 00195274 kdp_i386_trap (in kernel) + 276

0xffffff801e787544 : 00187544 kernel_trap (in kernel) + 1252

0xffffff801e61e1e0 : 0001e1e0 0xffffff800021e1e0 (in kernel) + 38

0xffffff801e66bc3c : 0006bc3c panic_trap_to_debugger (in kernel) + 540

0xffffff801e66b9fc : 0006b9fc panic (in kernel) + 92

0xffffff801ec6e2e1 : 0066e2e1 IOService::scheduleTerminatePhase2(unsigned int) (in kernel) + 529

0xffffff801ec71e17 : 00671e17 IOService::terminatePhase1(unsigned int) (in kernel) + 2311

0xffffff801ec916b1 : 006916b1 IOCatalogue::_terminateDrivers(OSDictionary*) (in kernel) + 161

0xffffff801ec918a5 : 006918a5 IOCatalogue::terminateDriversForModule(OSString*, bool) (in kernel) + 133

0xffffff801ec26d91 : 00626d91 OSKext::removeKext(OSKext*, bool) (in kernel) + 449

0xffffff801ec2cef5 : 0062cef5 OSKext::handleRequest(host*, unsigned int, char*, unsigned int, char**, unsigned int*, char**, unsigned int*) (in kernel) + 2085

0xffffff801ec3b9b4 : 0063b9b4 kext_request (in kernel) + 372

0xffffff801e6c1075 : 000c1075 0xffffff80002c1075 (in kernel) + 165

0xffffff801e6716e0 : 000716e0 ipc_kobject_server (in kernel) + 304

0xffffff801e64ea3d : 0004ea3d ipc_kmsg_send (in kernel) + 189

0xffffff801e66154b : 0006154b mach_msg_overwrite_trap (in kernel) + 891

0xffffff801e77171d : 0017171d mach_call_munger64 (in kernel) + 509

0xffffff801e61e9e6 : 0001e9e6 hndl_mach_scall64 (in kernel) + 22



Please look at the close code for the clinet



void userclient::stop(IOService* provider)

{

DEBUG("stop()\n");

if (fProvider) {

fProvider->close(this);

}


clientType = 0;


if (count == 0 && fProvider) {

fProvider->signal();

}

super::stop(provider);

}

IOReturn userclient::clientClose(void)

{

close();

terminate(0);

fProvider = NULL;

fTask = NULL;

return kIOReturnSuccess;

}


IOReturn userclient::close(void)

{

DEBUG("clientClose()\n");


if (clientType == dgaDaemon) {

agent_cleanup();

}

if(!fProvider)

return kIOReturnNotAttached;


if( fProvider && fProvider->isOpen(this) )

fProvider->close(this);



return kIOReturnSuccess;

}


bool userclient::terminate(IOOptionBits options)

{

if( fProvider && fProvider->isOpen(this) )

fProvider->close(this);

return super:terminate(options);

}

Replies

We are facing kernel panic while doing kextunload to our driver. This is working fine for the OSX <= 10.13.5.

On OSX >10.13.5 wile unloading the kext, its getting panic.

That’s only one macOS release greater than 10.13.5, and that’s 10.13.6. So, to be clear, it’s panicking on 10.13.6 but not panicking on 10.13.5?

Plese look at the below stack strace of the panic

That looks like an excerpt from the panic log. Can you post the full version?

ps If you mark it up as code (using the

<>
button) it’s easier to read.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

We have the exact situation. Was there any solution for this kernel panic?

thanks

Was there any solution for this kernel panic?

All I know about this issue is on this thread, that is, not much )-: If you can post a panic log, I’d be glad to take a look.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

See below.

This happens only with 10.13.5 and not all the time as well.

Out terminate function does not do much apart from calling to super::terminate().


thanks

P.

<code>

*** Panic Report ***

panic(cpu 0 caller 0xffffff800ec6e2e1): "com_ensilo_driver[0x1000007fd]::terminate(kIOServiceSynchronous) timeout\n"@/BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-4570.71.2/iokit/Kernel/IOService.cpp:2297



Backtrace (CPU 0), Frame : Return Address

0xffffff9249f1b810 : 0xffffff800e66c1c6

0xffffff9249f1b860 : 0xffffff800e795274

0xffffff9249f1b8a0 : 0xffffff800e787544

0xffffff9249f1b910 : 0xffffff800e61e1e0

0xffffff9249f1b930 : 0xffffff800e66bc3c

0xffffff9249f1ba60 : 0xffffff800e66b9fc

0xffffff9249f1bac0 : 0xffffff800ec6e2e1

0xffffff9249f1bb20 : 0xffffff800ec71e17

0xffffff9249f1bba0 : 0xffffff7f93bbe238

0xffffff9249f1bbc0 : 0xffffff800ec916b1

0xffffff9249f1bbf0 : 0xffffff800ec918a5

0xffffff9249f1bc30 : 0xffffff800ec26d91

0xffffff9249f1bc70 : 0xffffff800ec2cef5

0xffffff9249f1bcf0 : 0xffffff800ec3b9b4

0xffffff9249f1bd70 : 0xffffff800e6c1075

0xffffff9249f1bdc0 : 0xffffff800e6716e0

0xffffff9249f1be10 : 0xffffff800e64ea3d

0xffffff9249f1be60 : 0xffffff800e66154b

0xffffff9249f1bef0 : 0xffffff800e77171d

0xffffff9249f1bfa0 : 0xffffff800e61e9e6



Kernel Extensions in backtrace: com.ensilo.driver(3.0.1)[621B9AC1-0644-3FA0-9B49-A055ACDD35F7]@0xffffff7f93b82000->0xffffff7f949c2fff

BSD process name corresponding to current thread: kextunload



Mac OS version:

17G65



Kernel version:

Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64

Kernel UUID: 1AE5ACFD-3B6F-3D74-AD52-31F1430DBC6F

Kernel slide: 0x000000000e400000

Kernel text base: 0xffffff800e600000

__HIB text base: 0xffffff800e500000

System model name: MacBookPro14,3 (Mac-551B86E5744E2388)



System uptime in nanoseconds: 31038542827394

last loaded kext at 17557660098003: com.apple.driver.AppleXsanScheme 3 (addr 0xffffff7f949d8000, size 32768)

last unloaded kext at 18227837470126: com.apple.driver.AppleXsanScheme 3 (addr 0xffffff7f949d8000, size 32768)

loaded kexts:

com.google.dfsfuse.filesystems.dfsfuse 3.7.4

com.ensilo.driver 3.0.1 20

org.virtualbox.kext.VBoxNetAdp 5.2.16

org.virtualbox.kext.VBoxNetFlt 5.2.16

org.virtualbox.kext.VBoxUSB 5.2.16

org.virtualbox.kext.VBoxDrv 5.2.16

com.apple.filesystems.smbfs 3.2.3

com.apple.driver.AGPM 110.23.37

com.apple.driver.ApplePlatformEnabler 2.7.0d0

com.apple.driver.X86PlatformShim 1.0.0

com.apple.filesystems.autofs 3.0

com.apple.driver.AppleUpstreamUserClient 3.6.5

com.apple.kext.AMDFramebuffer 1.6.8

com.apple.kext.AMDRadeonX4000 1.6.8

com.apple.driver.AppleGraphicsDevicePolicy 3.20.13

com.apple.AGDCPluginDisplayMetrics 3.20.13

com.apple.driver.AppleHDA 281.52

com.apple.driver.AppleHV 1

com.apple.driver.AGDCBacklightControl 3.20.13

com.apple.driver.AppleHIDALSService 1

com.apple.iokit.IOUserEthernet 1.0.1

com.apple.kext.AMD9500Controller 1.6.8

com.apple.driver.AppleIntelKBLGraphics 10.3.6

com.apple.driver.AppleIntelPCHPMC 2.0.1

com.apple.iokit.IOBluetoothSerialManager 6.0.7f10

com.apple.driver.pmtelemetry 1

com.apple.Dont_Steal_Mac_OS_X 7.0.0

com.apple.driver.eficheck 1

com.apple.driver.AppleGFXHDA 100.1.17

com.apple.driver.AppleMCCSControl 1.5.5

com.apple.driver.SMCMotionSensor 3.0.4d1

com.apple.driver.AppleMuxControl 3.20.13

com.apple.driver.AppleOSXWatchdog 1

com.apple.driver.AppleEmbeddedOSSupportHost 1

com.apple.driver.AudioAUUC 1.70

com.apple.driver.AppleIntelKBLGraphicsFramebuffer 10.3.6

com.apple.driver.AppleIntelSlowAdaptiveClocking 4.0.0

com.apple.driver.AppleThunderboltIP 3.1.1

com.apple.filesystems.hfs.kext 407.50.6

com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1

com.apple.BootCache 40

com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0

com.apple.AppleSystemPolicy 1.0

com.apple.driver.AppleTopCaseHIDEventDriver 133

com.apple.driver.AirPort.BrcmNIC 1241.31.1a9

com.apple.filesystems.apfs 748.51.0

com.apple.driver.AppleSmartBatteryManager 161.0.0

com.apple.driver.AppleACPIButtons 6.1

com.apple.driver.AppleRTC 2.0

com.apple.driver.AppleSMBIOS 2.1

com.apple.driver.AppleACPIEC 6.1

com.apple.driver.AppleAPIC 1.7

com.apple.nke.applicationfirewall 183

com.apple.security.TMSafetyNet 8

com.apple.security.quarantine 3

com.apple.iokit.IOUSBUserClient 900.4.1

com.apple.kext.triggers 1.0

com.apple.kext.AMDRadeonX4100HWLibs 1.0

com.apple.kext.AMDRadeonX4000HWServices 1.6.8

com.apple.driver.DspFuncLib 281.52

com.apple.kext.OSvKernDSPLib 526

com.apple.iokit.IOAVBFamily 680.2

com.apple.plugin.IOgPTPPlugin 680.15

com.apple.iokit.IOEthernetAVBController 1.1.0

com.apple.iokit.IOBluetoothHostControllerUARTTransport 6.0.7f10

com.apple.iokit.IOBluetoothHostControllerTransport 6.0.7f10

com.apple.driver.AppleSSE 1.0

com.apple.driver.AppleHDAController 281.52

com.apple.iokit.IOHDAFamily 281.52

com.apple.driver.usb.IOUSBHostHIDDevice 1.2

com.apple.kext.AMDSupport 1.6.8

com.apple.driver.AppleBacklightExpert 1.1.0

com.apple.iokit.IONDRVSupport 519.20

com.apple.driver.AppleSMBusController 1.0.18d1

com.apple.driver.AppleIntelLpssUARTv1 3.0.60

com.apple.driver.AppleIntelLpssUARTCommon 3.0.60

com.apple.driver.AppleOnboardSerial 1.0

com.apple.iokit.IOSkywalkFamily 1

com.apple.iokit.IOSerialFamily 11

com.apple.driver.AppleGraphicsControl 3.20.13

com.apple.driver.X86PlatformPlugin 1.0.0

com.apple.driver.IOPlatformPluginFamily 6.0.0d8

com.apple.iokit.IOAcceleratorFamily2 378.26

com.apple.iokit.IOSurface 211.15

com.apple.AppleGPUWrangler 3.20.13

com.apple.AppleGraphicsDeviceControl 3.20.13

com.apple.iokit.IOGraphicsFamily 519.20

com.apple.iokit.IOSlowAdaptiveClockingFamily 1.0.0

com.apple.driver.AppleUSBAudio 312.6

com.apple.iokit.IOAudioFamily 206.5

com.apple.vecLib.kext 1.2.0

com.apple.driver.usb.cdc.ecm 5.0.0

com.apple.driver.usb.cdc.ncm 5.0.0

com.apple.driver.usb.AppleUSBiBridge 1.0

com.apple.driver.usb.cdc 5.0.0

com.apple.driver.usb.networking 5.0.0

com.apple.driver.usb.AppleUSBHostCompositeDevice 1.2

com.apple.driver.usb.AppleUSBHub 1.2

com.apple.driver.AppleUSBHostMergeProperties 1.2

com.apple.filesystems.hfs.encodings.kext 1

com.apple.driver.AppleActuatorDriver 1404.4

com.apple.driver.AppleHIDKeyboard 205

com.apple.driver.AppleHSBluetoothDriver 133

com.apple.driver.IOBluetoothHIDDriver 6.0.7f10

com.apple.iokit.IOBluetoothFamily 6.0.7f10

com.apple.driver.AppleMultitouchDriver 1404.4

com.apple.driver.AppleInputDeviceSupport 1404.3

com.apple.driver.AppleHSSPIHIDDriver 53

com.apple.driver.AppleHSSPISupport 53

com.apple.driver.AppleIntelLpssSpiController 3.0.60

com.apple.iokit.IO80211Family 1200.12.2

com.apple.driver.mDNSOffloadUserClient 1.0.1b8

com.apple.driver.corecapture 1.0.4

com.apple.driver.AppleThunderboltDPInAdapter 5.5.5

com.apple.driver.AppleThunderboltDPAdapterFamily 5.5.5

com.apple.driver.AppleThunderboltPCIDownAdapter 2.1.3

com.apple.iokit.IONVMeFamily 2.1.0

com.apple.driver.AppleThunderboltNHI 4.7.2

com.apple.driver.AppleHPM 3.1.3

com.apple.iokit.IOThunderboltFamily 6.7.8

com.apple.driver.AppleIntelLpssI2CController 3.0.60

com.apple.driver.AppleIntelLpssDmac 3.0.60

com.apple.driver.AppleIntelLpssGspi 3.0.60

com.apple.driver.AppleIntelLpssI2C 3.0.60

com.apple.driver.usb.AppleUSBXHCIPCI 1.2

com.apple.driver.usb.AppleUSBXHCI 1.2

com.apple.driver.usb.AppleUSBHostPacketFilter 1.0

com.apple.iokit.IOUSBFamily 900.4.1

com.apple.driver.AppleEFINVRAM 2.1

com.apple.driver.AppleEFIRuntime 2.1

com.apple.iokit.IOSMBusFamily 1.1

com.apple.iokit.IOHIDFamily 2.0.0

com.apple.security.sandbox 300.0

com.apple.kext.AppleMatch 1.0.0d1

com.apple.driver.DiskImages 480.60.1

com.apple.driver.AppleFDEKeyStore 28.30

com.apple.driver.AppleEffaceableStorage 1.0

com.apple.driver.AppleKeyStore 2

com.apple.driver.AppleUSBTDM 439.70.3

com.apple.driver.AppleMobileFileIntegrity 1.0.5

com.apple.iokit.IOUSBMassStorageDriver 140.70.2

com.apple.iokit.IOSCSIBlockCommandsDevice 404.30.2

com.apple.iokit.IOSCSIArchitectureModelFamily 404.30.2

com.apple.iokit.IOStorageFamily 2.1

com.apple.driver.AppleCredentialManager 1.0

com.apple.driver.KernelRelayHost 1

com.apple.iokit.IOUSBHostFamily 1.2

com.apple.driver.usb.AppleUSBCommon 1.0

com.apple.driver.AppleBusPowerController 1.0

com.apple.driver.AppleSEPManager 1.0.1

com.apple.driver.IOSlaveProcessor 1

com.apple.iokit.IOReportFamily 31

com.apple.iokit.IOTimeSyncFamily 680.15

com.apple.iokit.IONetworkingFamily 3.4

com.apple.driver.AppleACPIPlatform 6.1

com.apple.driver.AppleSMC 3.1.9

com.apple.iokit.IOPCIFamily 2.9

com.apple.iokit.IOACPIFamily 1.4

com.apple.kec.pthread 1

com.apple.kec.Libm 1

com.apple.kec.corecrypto 1.0



EOF



Model: MacBookPro14,3, BootROM MBP143.0178.B00, 4 processors, Intel Core i7, 2.9 GHz, 16 GB, SMC 2.45f0Graphics: Intel HD Graphics 630, Intel HD Graphics 630, Built-InGraphics: Radeon Pro 560, Radeon Pro 560, PCIeMemory Module: BANK 0/DIMM0, 8 GB, LPDDR3, 2133 MHz, 0x802C, 0x4D5435324C31473332443450472D30393320Memory Module: BANK 1/DIMM0, 8 GB, LPDDR3, 2133 MHz, 0x802C, 0x4D5435324C31473332443450472D30393320AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0x173), Broadcom BCM43xx 1.0 (7.77.37.31.1a9)Bluetooth: Version 6.0.7f10, 3 services, 27 devices, 1 incoming serial portsNetwork Service: USB 10/100/1000 LAN, Ethernet, en5USB Device: USB 3.0 BusUSB Device: USB2.0 HubUSB Device: SteelSeries Siberia EliteUSB Device: USB-C Digital AV Multiport AdapterUSB Device: iBridgeUSB Device: USB3.0 HubUSB Device: USB 10/100/1000 LANThunderbolt Bus: MacBook Pro, Apple Inc., 33.1Thunderbolt Bus: MacBook Pro, Apple Inc., 33.1

</code>

Thanks for the panic log. Looking at this it seems like the critical frame is this one:

0xffffff9249f1bac0 : 0xffffff800ec6e2e1

That frame denotes

IOService::scheduleTerminatePhase2(unsigned int)
. You can see the 10.13.3 source code for that routine in Darwin, with the specific panic being on line 2289. As you can see, I/O Kit’s termination logic is waiting for everything to shut down and, if that doesn’t happen within 15 seconds, it panics.

One thing to note here is that the Darwin code I linked to above is not the code you’re actually running. The panic log indicates you’re on xnu-4570.71.2, but the latest Darwin code is xnu-4570.50.219, and there have definitely been changes between those releases relevant to this code. Specifically, there has been at least one fix for exactly this sort of problem (r. 35752460).

As to why things are failing for you, I don’t have any specific input on that front. I do, however, have some general recommendations:

  • You should test on the latest 10.14 beta to see if you can reproduce the problem there.

  • If so, you should definitely file a bug about it. Please post your bug number, just for the record.

  • You should open a DTS tech support incident to see if we can find some sort of workaround.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

thanks for looking into that.

I saw that the panic is on timeout. What is not clear to me is what it is waiting for and if the mechanism has changed in 10.13.5?


teminate is the first step and it should wait only for work loop correct ?(https://developer.apple.com/library/archive/documentation/DeviceDrivers/Conceptual/IOKitFundamentals/DeviceRemoval/DeviceRemoval.html#//apple_ref/doc/uid/TP0000011-CH223-SW1)

What is not clear to me is what it is waiting for and if the mechanism has changed in 10.13.5?

It’s waiting for the service to become non-busy (that is,

kIOServiceBusyStateMask
to clear). This busy state is reflected in the public API via the
IOService::getBusyState()
and
IOService::adjustBusy()
methods. Those have doc comments that are well worth a read.

As to what’s making your service busy, it’s hard to say. With regards the bug I mentioned previously (r. 35752460), the underlying cause of the panic was a deadlock which stalled the thread which was keeping the service busy. That may well be the case here as well. And that may well be an OS bug or a bug in your KEXTs. It’s hard to say without doing a much deeper dive on this, and that’s going to be tricky in the context of DevForums.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"