OK, so wait, now is not the time. Thanks Quinn.

Cooked together a "notarytool" replacement that will do the work for me, and the build environment can remain the same. Instead of calling "xcrun notarytool" I run this: https://gist.github.com/lundman/9166dc8bef1973e5d9fc5428e0cedc57 Nothing special, but might save someone a few minutes having to do it themselves.

Same here, 10.14 and 10.15 both need to be notarized. With 10.15 I can copy the notarytool from Monterey's xcode (13.1), and it runs fine. With 10.14 it does not run due to: dyld: Library not loaded: /System/Library/Frameworks/CryptoKit.framework/Versions/A/CryptoKit If I copy over a local copy of CryptoKit, I can not set DYLD_LIBRARY_PATH anymore, sad-face. Possibly I could strip it of the codesign certificate and add "com.apple.security.cs.allow-dyld-environment-variables" but that feels like I'm close to yak shaving. 10.13 does not need notarizing, so it is odd Apple left one OS out in the cold. Has anyone come up with a solution that doesn't require me to copy it to a 2nd VM running newer macOS?

Word is that there now is a work-around, if somewhat complicated. I suspect with a bit more time, it will be streamlined down to something nicer.

This is still an issue, including on the sonoma release. Snapshot API calls are there, but the ability to load kexts is still missing.

Possibly it can be done, but I went to beta3 instead, which has a matching KDK.

To be complete, this is what we went with: #if defined(__aarch64__) uint64_t mpidr_el1; asm volatile("mrs %0, mpidr_el1" : "=r" (mpidr_el1)); /* * To save us looking up number of eCores and pCores, we * just wrap eCores backwards from max_ncpu. * 0: [P0 P1 P2 ... Px Ex .. E2 E1 E0] : max_ncpu * * XNU: Aff2: "1" - PCORE, "0" - ECORE */ #define PCORE_BIT (1ULL << 16) if (mpidr_el1 & PCORE_BIT) return ((uint32_t)mpidr_el1 & 0xff); else return ((max_ncpus -1) - (uint32_t)(mpidr_el1 & 0xff)); #else return ((uint32_t)cpu_number()); #endif }

uint32_t getcpuid(void) { #if defined(__aarch64__) uint64_t mpidr_el1; asm volatile("mrs %0, mpidr_el1" : "=r" (mpidr_el1)); return (uint8_t)(mpidr_el1 & 0xff); #else return ((uint32_t)cpu_number()); #endif } Is a start, on my 8core M2, it returns values 0x00 to 0x03 from Aff0 bits. Generally the 4 fast cores. If you idle for a bit, you can probably look at the Aff1 (cluster) values, returning values like 0x0102 - which probably means the slower cores. But for the sake of reducing contention by spreading out locks over cores, this is a good start.

With exhaustive printf debugging - since this is now the best available, we found out the old line: static char initial_default_block[16ULL*1024ULL*1024ULL] __attribute__((aligned(16384))) = { 0 }; Used to bootstrap our memory allocator, would not work on Ventura+M1+more-than-16GB ram. Ie, MacStudio 128G. Monterey on same hardware (As well as all lower M1s, and x86_64) are fine. We replaced it with: initial_default_block = IOMallocAligned(INITIAL_BLOCK_SIZE, 16384); memset(initial_default_block, 0, INITIAL_BLOCK_SIZE); Unlikely anyone else would bump into this issue, but it feels more complete to post the answer.

Maybe you could write a program that will dlopen() and 32bit app, the intercept any interlibrary calls, libc, syscall, with glue to translate 32bit to 64bit and back. It would be a massive undertaking, with no guarantee that you wouldn't hit a brick wall at somepoint.. .. when all "they'd" have to do is compile it for 64 bit :) Might even be easier to go translate the 32bit assembler to 64bit in each segment. But that does nothing to help you with the "old" API calls, you'd still need to interpose them with glue.

The KDK readme says: Note: Apple silicon doesn’t support installing the kernel and kernel extension variants from the KDK. So I think you just can't, even now, three major versions later.

OK, dtracing the debug kernel helped here, as certain functions were not inlined: dtrace -n 'mac*:entry / execname == "nfsd" / {} ' -n 'mac*:return / execname == "nfsd" / { printf("%d %s", arg1, execname); }' -n 'nfs*:entry / execname == "nfsd" / {} ' -n 'nfs*:return / execname == "nfsd" / { printf("%d %s", arg1, execname); }' -n 'vnode*:entry / execname == "nfsd" / {} ' -n 'vnode*:return / execname == "nfsd" / { printf("%d %s", arg1, execname); }' -n 'vn*:entry / execname == "nfsd" / {} ' -n 'vn*:return / execname == "nfsd" / { printf("%d %s", arg1, execname); }' -n 'nfsrv_rephead:entry { nd = arg0 + 0x88; tracemem(nd, 4); printf("rephead %d", arg0);}' Yeah, the whole thing. Anyway, it produced this hint: 1 269114 mac_vnode_check_open:entry 1 269042 mac_cred_check_enforce:entry 1 269043 mac_cred_check_enforce:return 1 nfsd 1 232364 vn_getpath_ext_with_mntlen:entry 1 231850 build_path_with_parent:return 2 nfsd 1 232365 vn_getpath_ext_with_mntlen:return 2 nfsd 1 232364 vn_getpath_ext_with_mntlen:entry 1 231850 build_path_with_parent:return 2 nfsd 1 232365 vn_getpath_ext_with_mntlen:return 2 nfsd 1 268848 mac_error_select:entry 1 268849 mac_error_select:return 2 nfsd 1 269115 mac_vnode_check_open:return 2 nfsd ie, it's not that mac_vnode_check_open() but rather that vn_getpath_ext_with_mntlen() fails. (Technically, build_path_with_parent()). At the end of my zfs_vnop_create() - if I add the call: vnode_update_identity(*ap->a_vpp, NULL, (const char *)ap->a_cnp->cn_nameptr, ap->a_cnp->cn_namelen, 0, VNODE_UPDATE_NAME); I get a successful test run using O_EXCL over nfs, finally. I am not entirely sure why this is required, only place I call vnode_update_identity() is in vfs_vget(). Otherwise, I've "been getting away" with not calling it all this time. I assume it doesn't hurt to call it.

I'd be happy to not know it, if you know what I mean. But since NFS+O_EXCL isn't working right here, and yet, is working correctly with HFS, I just don't see why yet. HFS does have a handful of vnode_authorize() calls, of which I have zero. That is my current thinking anyway. There is a bunch of debug stuff in MAC, is it trivial to turn on to perhaps see why it complains? Running kernel.debug 10.15.7 Thanks for replying,

Running kernel.debug and dtrace - it looks like: 1 1 nfsrv_setattr:entry 1 1 mac_vnode_check_open:entry 1 1 mac_cred_check_enforce:return 1 nfsd 1 1 mac_label_get:return 0 nfsd 1 1 mac_error_select:return 2 nfsd 1 1 mac_error_select:return 2 nfsd 1 1 mac_error_select:return 2 nfsd 1 1 mac_policy_list_conditional_busy:return 0 nfsd 1 1 kernel.debug`nfssvc_nfsd+0x108b kernel.debug`nfssvc+0x360 kernel.debug`unix_syscall64+0x81f kernel.debug`hndl_unix_scall64+0x16 1 1 mac_vnode_check_open:return 2 nfsd 1 1 nfsrv_rephead:entry 0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef 0: 46 00 00 00 F... Which makes me thing that after the call to mac_label_get() it checks something and is not happy. I have not really looked into what vnode_label is but presumably has something to do with auth.

It is curious as to why nfs server figures out that exclusive is set, then clears va_mode? Oh yeah, this came screaming back to me. So when NFS has to do exclusive, it attempts to create the file with metadata cleared, that is va_mode 0, and time. It uses atime to hold a create_verf (probably IP and counter), then uses getattr() to confirm the create_verf is matching (this client won). (va_flags has VA_UTIMES_NULL). If that goes well, it calls setattr() with the correct information. # We are about to create the file, with O_EXCL, no existing file. 0 nfsrv_create:entry 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd # Which finally calls our vnop_create 0 zfs_vnop_create:entry # we have created it! 0 zfs_vnop_create:return 0 nfsd # nfsrv_create now calls setattr - probably to set atime 0 zfs_vnop_setattr:entry 0 zfs_vnop_setattr:return 0 nfsd # done, and some getattr verify 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd # send reply code to client 0 nfsrv_rephead:entry 0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef 0: 00 00 00 00 .... # nfsrv_create returns success! We have created the file 0 nfsrv_create:return 0 nfsd # this setattr appears to be the next client request, unsure where it is from # but probably to set mode/uid/gid 0 nfsrv_setattr:entry 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd # nfsrv_setattr calls mac_vnode_check_open 0 mac_vnode_check_open:entry 0 zfs_vnop_getattr:entry 0 zfs_vnop_getattr:return 0 nfsd 0 zfs_vnop_getattr:entry 1 zfs_vnop_getattr:return 0 nfsd # this setattr is about to fail. 1 hook_vnode_check_open:return 2 nfsd 1 hook_vnode_check_open:return 0 nfsd 1 vng_vnode_check_open:return 0 nfsd 1 mac_vnode_check_open:return 2 nfsd 1 mac_vnode_check_open:return 2 nfsd # mac_vnode_check_open failed, error = ESTALE 1 zfs_vnop_getattr:entry 1 zfs_vnop_getattr:return 0 nfsd 1 nfsrv_rephead:entry 0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef 0: 46 00 00 00 F... I guess if I knew where hook_vnode_check_open() is defined, I can perhaps figure out what goes wrong, but I get a bit lost in the MACF macros.