zpool binary, and all its .dylibs, are signed and notarized. Hopefully correctly, but let's leave a little room for the possibility I got something wrong.I need to think up a good way to test the issue, since it appears to trigger at boot (when it is supposed to import the pools), from launchd. I believe when run by hand after a user logged in, all is well. Could it be running too early, before the OS has a chance to verify certificates?I found I had to leave "mount_zfs" unsigned for mount_root - or it would fail to mount root at boot time as it was unable to verify the teamID. But that is unrelated I think.

# mv python pythonold# ln -s ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 python# sw_versProductName: Mac OS XProductVersion: 10.14.6BuildVersion: 18G4032# which python/usr/bin/python# ls -l /usr/bin/pythonlrwxr-xr-x 1 root wheel 75 May 5 06:31 /usr/bin/python -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7# python --versionPython 2.7.16# lldb /Library/Developer/KDKs/KDK_10.14.6_18G2016.kdk/System/Library/Kernels/kernel(lldb) target create "/Library/Developer/KDKs/KDK_10.14.6_18G2016.kdk/System/Library/Kernels/kernel"unable to load scripting data for module kernel - error reported was Missing parentheses in call to 'print'. Did you mean print("Loading kernel debugging from %s" % __file__)? (kernel.py, line 69) File "temp.py", line 1, in <module>It is stubborn alright.

It seems rather determined to run python3:# /usr/bin/lldb -P/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python3# /usr/bin/lldb --versionlldb-1100.0.30.12Apple Swift version 5.1.3 (swiftlang-1100.0.282.1 clang-1100.0.33.15)However:# /Library/Developer/CommandLineTools/usr/bin/lldb -P/Library/Developer/CommandLineTools/Library/PrivateFrameworks/LLDB.framework/Resources/Python# /Library/Developer/CommandLineTools/usr/bin/lldb --versionlldb-1001.0.13.3 Swift-5.0# /Library/Developer/CommandLineTools/usr/bin/lldb /Library/Developer/KDKs/KDK_10.14.6_18G2016.kdk/System/Library/Kernels/kernel(lldb) target create "/Library/Developer/KDKs/KDK_10.14.6_18G2016.kdk/System/Library/Kernels/kernel"Loading kernel debugging from /Library/Developer/KDKs/KDK_10.14.6_18G2016.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/Python/kernel.pyLLDB version lldb-1001.0.13.3 Swift-5.0settings set target.process.python-os-plugin-path "/Library/Developer/KDKs/KDK_10.14.6_18G2016.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/Python/lldbmacros/core/operating_system.py"settings set target.trap-handler-names hndl_allintrs hndl_alltraps trap_from_kernel hndl_double_fault hndl_machine_check _fleh_prefabt _ExceptionVectorsBase _ExceptionVectorsTable _fleh_undef _fleh_dataabt _fleh_irq _fleh_decirq _fleh_fiq_generic _fleh_deccommand script import "/Library/Developer/KDKs/KDK_10.14.6_18G2016.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/Python/lldbmacros/xnu.py"xnu debug macros loaded successfully. Run showlldbtypesummaries to enable type summaries.settings set target.process.optimization-warnings falseSo I have a way to keep running KDKs.. Sorry for the noise.

/usr/bin/lldb fail, but with additional messageWARNING! Python version 3 is not supported for xnu lldbmacros.Please restart your debugging session with the following workarounddefaults write com.apple.dt.lldb DefaultPythonVersion 2So that is all the clues one needs maybe, using the /Library/Developer/CommandLineTools/usr/bin/lldbeverything works like before.

Not sure I can call a user-space-mounting API from kernel though.I'd like to call "kernel_mount()" but it is private, and only for NFS. Failing that, __mac_mount() but also private. Then there is the userland handler mount() - but it takes a "struct mount_args" which isn't defined, unless I shadow it - but I was hoping to find a "legal" way to mount from kernel.

I do believe the proper way is to create a new device in kernel, with ContentHint, and have a fs-bundle match it in userland, diskarbitrationd kicks in, have a utility for probe, and utility to issue mount, then we come back into the kernel for mounting - yes.That is a lot of work to avoid calling "kernel_mount()" but presumably that is what Apple wants us to do. Except, their one open-source example, NFS, calls kernel_mount like a first class citizen, so they aren't setting a very good example. So I wanted to see if Apple's new snapshot-mount they added very recently would allow us to trigger a snapshot mount from kernel.I'm doing it in the kernel for the same reason NFS is. So I look forward to what Apple does to NFS so I can follow suit.

IMO NFS is not a good example of anything (-: Hah, it is showing its age, that is true. Thanks for letting me know what I suspected.What would have been awesome though, is if there was an Apple approved example. 🙂Cheers,Lund

What, you mean like MFSLives?Never seen it before, and I spent hours googling for it back in 2013. But that aside, I only see userland calling mount(), and notthe kernel triggered a mount - but I did only skim over the sources.Seriously though, it’s hard to recommend that anyone develop a VFS plug-in at this point. Apple has already announced that we plan to discontinue KEXTs at some point in the future.Of course, this is work started in 2013, so I'll keep supporting it until Apple kicks us out, then I'll have to find another platform (or hobby).on iOS, we have File Provider extensionsAnd do you think that is something good enough to do a full ZFS? Or even EXT3/NTFS etc? I had a bunch of questions I was curious about wrt to XNU kernel, but I suppose there isn't much reason to chase those down 🙂

File providers offer a very different view of the world. There’s nothing stopping you from using them to implement ZFS, but the end result would be very different from the VFS plug-in approach that you’re used to True - but interesting you didn't immediately say "impossible". I recently ported ZFS to Windows, and that sure was different to a VFS style system.Cheers,Lund

Ok well, you may not want to see this; So there are a bunch of functions we desperately need to call to be able to function (38 to date), so we produce a "Codeless Kernel Extension" to help us link, like; $(NM) -gj $(MACH_KERNEL) > allsymbols ./kextsymboltool -arch x86_64 -import allsymbols -export zfs.exports -output kernelexports_64 lipo -create kernelexports_32 kernelexports_64 -output kernelexports cp kernelexports zfs.kext/Contents/Plugins/KernelExports.kext/KernelExports then tie the main kext to depend on it, and voila. Alas, some changes to "kextsymboltool" must have happened for BigSur, so the version in Catalina no longer produces something that BigSur will load. I guess I was hoping for "kextsymboltool.c", but that's probably too soon, so maybe a binary of "kextsymboltool". Ultimately, we'd love to get rid of the 38 items in zfs.exports file, but not found solutions for the issues they solve - but that is a whole other thing, and will probably become moot when kexts are removed entirely.

Oh I was reading: https://developer.apple.com/library/archive/documentation/Darwin/Conceptual/KernelProgramming/vm/vm.html the section with "Allocating Memory In the Kernel Itself". But it is probably old news. Looks like I can call OSMalloc, with our own-allocator with quantum set over kalloc_kernmap_size. I will try that first. While I have your attention (do I though?) - it would be nice if we could use xnu's kalloc.zones, but we struggled with it for quite a long time. The issue was always that the machine would simply panic, if it ran out of a zone. (say, zone.64) Surely there is a way to be told/warned/event that a zone is getting full, or under pressure? Would really like to avoid panic, even if it means stalling an allocation long enough for it to reap. We never found a memory-pressure system we were allowed to call, (or it was calling us After it spun looking for memory and calling panic).

Oh huh, I *assumed* it does not fail on hfs and apfs, but turns out it does. And ssh localhost will make it not-fail. I get Operation not permitted both hfs and apfs can give chown -R error. both hfs and apfs will not give error if you ssh localhost.

Oh ah, i see. If I add "Full Disk Access" to "Terminal" (sshd already has it) then it works. That isn't all that exciting then. Ah well, on to next mystery...

Ah ok, my bad, it didnt pay attention to boot-args as I'd "cdrutil enable --without nvram" when KDK says just to disable. Not sure if either works completely, as I have no wired network for MBP yet. But the symbolication question is what I am hoping to have answered, like when users send you panic reports.

One of the users noticed something interesting; objdump -S /Library/Extensions/zfs.kext/Contents/MacOS/zfs|less 0000000000088000 _abd_verify: Using the first symbols offset, 0x88000, the stack posted in the panic makes sense. So change load address 0xfffffe002641c000 - 0x88000 = 0xFFFFFE0026394000. atos -o /Library/Extensions/zfs.kext/Contents/MacOS/zfs -arch arm64e -l 0xFFFFFE0026394000 0xfffffe00266449d4 0xfffffe002650ab60 0xfffffe002650fad4 0xfffffe002650dc88 0xfffffe0026517798 0xfffffe002763f82c getf (in zfs) (spl-vnode.c:218) zfs_file_get (in zfs) (zfs_file_os.c:412) zfs_ioc_send_new (in zfs) (zfs_ioctl.c:6472) zfsdev_ioctl_common (in zfs) (zfs_ioctl.c:7603) zfsdev_ioctl (in zfs) (zfs_ioctl_os.c:234) Similarly, lldb can be made to work with offsets - 0x88000, as loaded from 0. Not when using load address. The actual panic? Called vnode_specrdev() on a vnode not CHR/BLK.