Hi, It seems that on M4 devices any virtual machine with macOS version older than 13.4 fail to boot, they stuck with a black screen. This is regardless of the virtualization software used (UTM, VirtualBuddy, Viable, etc...). After talking to many people everyone experiences the same. At least for me, this is a massive limitation of the platform, I really hope this is a bug which can be fixed. Thanks, Csaba

https://feedbackassistant.apple.com/feedback/15645457 Metal passthrough on intel VMs causes com.apple.screensharing.menuextra to crash and screensharing to exit Create a 15.1 VM with metal passthrough on 15.0.1 or 15.1 host, enable Screen Sharing, then try connecting to with VNC after restarting the machine. I'm using Anka to create the VM. You'll see VNC work (open vnc://192.168.64.3:5900), then a few seconds in show "Reconnecting...", then work, then go to "Reconnecting..." for ~5m until it eventually works consistently. You'll see launchd showing exits/failures (see screenshots) You'll see diagnostic reports showing things like: Thread 0 Crashed:: Dispatch queue: com.apple.RenderBox.Encoder 0 libsystem_kernel.dylib 0x7ff801da5b52 __pthread_kill + 10 1 libsystem_pthread.dylib 0x7ff801ddff85 pthread_kill + 262 2 libsystem_c.dylib 0x7ff801d00b19 abort + 126 3 libsystem_c.dylib 0x7ff801cffddc __assert_rtn + 314 4 Metal 0x7ff80d045d72 MTLReportFailure.cold.1 + 41 5 Metal 0x7ff80d01fa2a MTLReportFailure + 513 6 Metal 0x7ff80cfb74e0 +[MTLLoader sliceIDForDevice:legacyDriverVersion:airntDriverVersion:] + 200 7 Metal 0x7ff80cf265c9 +[_MTLBinaryArchive(MTLBinaryArchiveInternal) deserializeBinaryArchiveHeader:fileData:device:] + 89 8 Metal 0x7ff80cf10f0c -[_MTLBinaryArchive loadFromURL:error:] + 537 9 Metal 0x7ff80cf10288 -[_MTLBinaryArchive initWithOptions:device:url:error:] + 844 10 RenderBox 0x7ff9041a15fd RB::(anonymous namespace)::load_library_archive(NSBundle*,

I have a MacMini M2 machine running Sequoia 15.1 OS. On this machine, I am running a Virtual Machine, utilizing the Virtualization.Framework, with the same OS version, 15.1. Logging into my account in the System Settings is successful. Next, I need to add my account in Xcode 16.1. While the initial login is successful, Xcode immediately displays the following error: Decoding Error. There was a failure decoding response: (HTTP 401, 60 bytes) The data couldn’t be read because it isn’t in the correct format. As a result, I cannot see any account information, teams, etc. A very similar bug has been reported at this issue - https://developer.apple.com/forums/thread/759877, but there has been no progress or updates there. Is there any chance to fix this and get it working?

Hi! I would like try to boot the Linux kernel with the Hypervisor framework and see how far I get. So far the kernel runs up to the point where it's trying to identify the redistributor of the Hypervisor's GICv3, but I get an exception when it's reading the memory-mapped GICR_FIDR2 register. I tried the same via hv_gic_get_redistributor_reg() and get HV_DENIED. What could be the reason for this exception? I believe I've initialized enough of the GIC for it to work. No interrupts yet, though. It is of course entirely possible I forgot to set/clear some bits, but there are several redistributor registers missing in the framework, so it's not possible to do the full initialization a hardware GIC v3 implementation needs. I assume the Hypervisor's GIC abstraction takes care of several steps internally. What are the steps to initialize the HVF's GIC? Do you have a working example? I couldn't find anything on the internet. The popular virtualization software out there all seem to bring their own emulated interrupt controller. I'm using Sequoia 15.0.1. Thank you for any hints!

Hi, I'm building a Virtual Machine Manager on top of Hypervisor Framework and having a problem with hv_vcpu_run never return somehow. I tested the same code on Asahi Linux using KVM and everything are working correctly. I really have no idea what I'm doing wrong here. How I setup a vCPU: https://github.com/obhq/obliteration/blob/main/gui/src/vmm/aarch64.rs How I use the Hypervisor Framework: https://github.com/obhq/obliteration/blob/main/gui/src/vmm/hv/macos/cpu.rs#L402 Thanks in advance.

Introduction I'm trying to integrate support for the APIC implementation added to Hypervisor.framework back in macOS 12 into the open source Qemu VMM. Qemu contains a VMM-side software implementation of the APIC, but it shows up as a major performance constraint in profiling, so it'd be nice to use the in-kernel implementation. I've previously submitted DTS TSIs (case 3345863) for this and received some high level pointers, but I'm told the forums are now the focus for DTS. I've got things working to what feels like 95%, but I'm still tripping up on a few things. FreeBSD and macOS guests are successfully booting and running most of the time, but there are sporadic stalls which point towards undelivered interrupts. Linux fails early on. A number of key test cases are failing in the 'apic' and 'ioapic' test suites that are part of the open source 'kvm-unit-tests' project, and I've run out of ideas for workarounds. Broadly, I'm doing this: When calling hv_vm_create, I pass the HV_VM_ACCEL_APIC flag. The VM uses the newer hv_vcpu_run_until() API. After VM exits, query hv_vcpu_exit_info() in case there's anything else to do. Page fault VM exits in the APIC's MMIO range are forwarded to hv_vcpu_apic_write and hv_vcpu_apic_read respectively. (With hv_vcpu_exit_info check and post-processing if no_side_effect returns true.) Writes to the APICBASE MSR do some sanity checks (throw exception on invalid state transitions etc) and update the MMIO range via hv_vmx_vcpu_set_apic_address() if necessary. HVF seems to do its own additional handling for the actual APIC state changes. (Moving the MMIO and enabling the APIC at the same time fails: FB14021745) Various machinery and state handling around INIT and STARTUP IPIs for bringing up the other vCPUs. This was fiddly to get working but I think I've got it now. MSIs from virtual devices are delivered via hv_vm_lapic_msi. Reads and writes for PIC and ELCR I/O ports are forwarded to the hv_vm_atpic_port_write/hv_vm_atpic_port_read APIs. (In theory, interrupt levels on the PIC are controlled via hv_vm_atpic_assert_irq/hv_vm_atpic_deassert_irq but all modern OSes disable the PIC anyway.) Page faults for the IOAPIC's MMIO range are forwarded to hv_vm_ioapic_read/hv_vm_ioapic_write. Virtual devices deliver their interrupts using hv_vm_ioapic_assert_irq/hv_vm_ioapic_deassert_irq/hv_vm_ioapic_pulse_irq for level/edge-triggered interrupts respectively. Now for the parts where I'm stuck and I'm either doing something wrong, or there are bugs in HVF's implementation: Issues I'm running into IOAPIC: 1. Unmasking during raised interrupt level, test_ioapic_level_mask test case: Guest enables masking on a particular level-triggered interrupt line. (MMIO write to ioredtbl entry) The virtual device raises interrupt level to 1. VMM calls hv_vm_ioapic_assert_irq(). No interrupt, because masked, so far so good. The guest unmasks the interrupt via another write to the ioredtbl entry. At this point I would expect the interrupt to be delivered to the vCPU. This is not the case. Even another call to hv_vm_ioapic_assert_irq() after unmasking will have no effect. Only if we deassert and reassert does the guest receive anything. (This is my current workaround, but it is rather ugly because I essentially need to maintain shadow state to detect the situation.) 2. Retriggering, test case test_ioapic_level_retrigger: The vCPU enters a interrupts-disabled section (cli instruction) The virtual device asserts level-triggered interrupt. VMM calls hv_vm_ioapic_assert_irq(). The vCPU leaves the interrupts-disabled section (sti instruction) and starts executing other code (or halts, as in the test case) Interrupt is delivered to vCPU, runs interrupt handler. Interrupt handler signals EOI. Note that interrupt is still asserted. Outside the interrupt handler, the vCPU briefly disables interrupts again (cli) The vCPU once again re-enables interrupts (sti) and halts (hlt) Here we would expect the interrupt to be delivered again, but it is not. I don't currently have a workaround for this because none of these steps causes hv_vcpu_run_until exits where this condition could be detected. 3. Coalescing, test_ioapic_level_coalesce: The virtual device asserts a level-triggered interrupt line. The vCPU enters the corresponding handler. The device de-asserts the interrupt level. The device re-asserts the interrupt. The device once again de-asserts the interrupt. The interrupt handler sets EOI and returns. We would expect the interrupt handler to only run once in this sequence of events, but as it turns out, it runs a second time! This is less critical than the previous 2 unexpected behaviours, because spurious interrupts are usually only slightly detrimental to performance, whereas undelivered interrupts can cause system hangs. However, this doesn't exactly instill confidence in the implementation. I've submitted the above, as they look like bugs to me - either in the implementation, or lack of documentation - as FB14425412. APIC To work around the HVF IOAPIC problems mentioned above, I tried to use the HVF APIC implementation in isolation without the ATPIC and IOAPIC implementations. Instead, I provided (VMM side) software implementations of these controllers. However, the software IOAPIC needs to receive end-of-interrupt notifications from the APIC. This is what I understood the HV_APIC_CTRL_IOAPIC_EOI flag to be responsible for, so I passed it to hv_vcpu_apic_ctrl() during vCPU initialisation. The software IOAPIC implementation receives all the MMIO writes, maintains IOAPIC state, and calls hv_vm_send_ioapic_intr() whenever interrupts should be delivered to the VM. However, I have found that hv_vcpu_exit_info() never returns HV_VM_EXITINFO_IOAPIC_EOI. When the HVF APIC is in xAPIC mode, I can detect writes to offset 0xb0 in the MMIO write handler and query hv_vcpu_exit_ioapic_eoi() for the vector whose handler has run. However, once the APIC is in x2APIC mode, there are no exits for the x2APIC MSR accesses, so I can't see how I might get those EOI notifications. Am I interpreting the purpose of HV_APIC_CTRL_IOAPIC_EOI correctly? Do I need to do anything other than hv_vcpu_apic_ctrl to make it work? How should I be receiving the EOI notifications? I was expecting vCPU run exits, but this does not appear to be the case? Again, either a crucial step is missing from the documentation, or there's a bug in the implementation. I've submitted this as FB14425590. My Questions: Has anyone got the HVF APIC/IOAPIC working for the general purpose case, i.e. guest OS agnostic, all edge cases handled? The issues I've run into - are these bugs in HVF? Do I need extra support code/workarounds to make the edge cases work? Is using the APIC without the HVF's IOAPIC an intended supported use case or am I wasting my time on this "split" setup?

Last year I remember Apple providing a Device Support download that would allow running macOS 14 in a VM on macOS 13 without needing to install the whole Xcode beta. Is Apple planning on releasing a similar download this year to allow running a macOS 15 VM on macOS 14 without installing the Xcode beta?

Is there any official way to download IPSW files for older versions of macOS like macOS 13 for use with a VM app like Parallels or UTM?

The M2-series and M3-series of processors seem to use the same ARMv8.6-A extension, however nested virtualization is restricted to the M3-series chips. Is there an architectural reason why this is?

Hello, I have tried to create a VZVirtualMachine macOS virtual machine on an Apple Silicon Mac. I have installed Docker Desktop inside this virtual machine. Docker is not working: I get this error message: "Hyopervisor check failed". Is there a way to run Docker inside a VZVirtualMachine ? Thanks a lot

Still no hypervisor support in iPadOS 17 Hypervisor is indeed physically possible on any of the M series chips included in the iPad Airs and iPad Pros, but locked away the iPadOS. Block hypervisor on iOS is reasonable to me, because it consume powers, not frendly for battery and not sutable for a mobile phone. But for iPadOS, the limitation is not reasonable to me. First, the Guideline 2.5.2 of iOS and iPadOS blocks code execution that loads dynamically, it may protect users because apps may load malicious code after it passes the revew from app store. But if we load codes in the hypervisor, any malicious can only run in the VM, and the safety of the VM is not an issue. Escape from a VM is even harder than escape from the sandbox of the safari browser. Even there are still other concerns about load arbitrary codes to hypervisor, we can limit it only load user selected code to the hypervisor, blocks app load code from interent without user intention. Running user selected code in the hypervisor won't threaten the security at all, there is no reason for Guideline 2.5.2 applies to hypervisor. Second, iPad a laptop replacement in the advertisement. As a laptop, it can't execute any user generated code on it, it can only be interpreted. As a software develper, it means iPadOS basically not useable. I can only run code on a remote server, and use iPad as a thin client. It can't be a standalone devices, even it has a powerful M2 chip. For the xcode on iPad, if apple concerns xcode on iPad breaks the security model, we can run the compiled code in the hypervisor, which isolates the reviewed code and user generated code. iPad has a powerful M2 chip, but iPadOS limit the power for it.

How does VMWare access USB devices without have any specifics of the USB device? Does it use the same profile/entitlement process or does it take a different approach?

Hi. I want to use 4k-aligned page mapping through the hv_vm_* apis. But my mac's page size is 16k, so it would occur an error when I do that. Is there a way to change the permission of a 4k-aligned guest memory? Thank you.

I use UTM.app for virtualisation. I have full virtualise "Fedora 38-aarch64" in UTM.app with rosetta enabled. After upgrading Sonoma to 14.3 it stop properly virtualised shared memory. I have this test file: #include <stdio.h> #include <sys/shm.h> #include <sys/stat.h> int main () { int segment_id; char* shared_memory; struct shmid_ds shmbuffer; int segment_size; const int shared_segment_size = 0x6400; /* Allocate a shared memory segment. */ segment_id = shmget (IPC_PRIVATE, shared_segment_size, IPC_CREAT | IPC_EXCL | S_IRUSR | S_IWUSR); /* Attach the shared memory segment. */ shared_memory = (char*) shmat (segment_id, 0, 0); printf ("shared memory attached at address %p\n", shared_memory); /* Determine the segment's size. */ shmctl (segment_id, IPC_STAT, &shmbuffer); segment_size = shmbuffer.shm_segsz; printf ("segment size: %d\n", segment_size); /* Write a string to the shared memory segment. */ sprintf (shared_memory, "Hello, world."); /* Detach the shared memory segment. */ shmdt (shared_memory); /* Reattach the shared memory segment, at a different address. */ shared_memory = (char*) shmat (segment_id, (void*) 0x5000000, 0); printf ("shared memory reattached at address %p\n", shared_memory); /* Print out the string from shared memory. */ printf ("%s\n", shared_memory); /* Detach the shared memory segment. */ shmdt (shared_memory); /* Deallocate the shared memory segment. */ shmctl (segment_id, IPC_RMID, 0); return 0; } Command to compile it is gcc -Wall a.c && ./a.out When I compile it in virtualised Fedora work properly show this: shared memory attached address segment size: shared memory reattached address Hello, world. When I compile directly on M1 mac id it die shared memory attached address segment size: shared memory reattached address Segmentation fault: I'm try it also in docker x86 in virtualised fedora and also show error In "Fedora 38-aarch64 virtualised" run x86 docker "docker run -it --platform linux/amd64 oraclelinux:7.9 bash" Install gcc in docker shell "yum install -y gcc" After compile and run it die with shared memory attached address segment size: shared memory reattached address Hello, world. assertion failed [rem_idx != ]: Unable find existing allocation shared memory segment to unmap (VMAllocationTracker.cpp remove_shared_mem) Trace/breakpoint (core dumped) How can I fix it? On previous version of Sonoma works properly. Thank you

I used to run VirtualBox on macOS to run Windows guests for some reasons. Recently I bought a new Mac mini M1, now I have a problem - VB does not have a stable release for arm (yet). What other options do I have? BTW, I came across this doc article (https://developer.apple.com/documentation/virtualization/running_macos_in_a_virtual_machine_on_apple_silicon). I read thru it, but could not conclude if it offers the same functionalities as a full-blown VM suite; and more specifically I want to run Windows guests.

Hi, everyone! I want to know whether I can check my app is running on a virtual machine or not. If so, how can I do that in C++ or Objective-C? The code needs to support both Intel processors and Apple Silicon. Thanks! Leo

I am following the Running Linux in a Virtual Machine guide on MacOS 14 w/ ARM. Fedora kernel and RAM disk images were acquired for aarch64. I opened LinuxVirtualMachine.xcodeproj from the example; selecting 'LinuxVirtualMachine' under 'Targets', and navigating to 'Signing and Capabilities', I linked my Personal Team (needed to connect my Apple ID) under 'Signing->Team'. Clicking 'run' outputs the following: Usage: /Users/shea/Library/Developer/Xcode/DerivedData/LinuxVirtualMachine-baalsbzvoxxbspgicozzllxbyqmh/Build/Products/Debug/LinuxVirtualMachine <kernel-path> <initial-ramdisk-path> Program ended with exit code: 64 So I execute the following in Terminal from a directory containing the two images from Fedora: % /Users/shea/Library/Developer/Xcode/DerivedData/LinuxVirtualMachine-baalsbzvoxxbspgicozzllxbyqmh/Build/Products/Debug/LinuxVirtualMachine ./vmlinuz ./initrd.img And receive the output: Failed to start the virtual machine. Error Domain=VZErrorDomain Code=1 "The virtual machine failed to start." UserInfo={NSLocalizedFailure=Internal Virtualization error., NSLocalizedFailureReason=The virtual machine failed to start.} An old thread seems to imply this may be because the kernel image used to be compressed but is no longer? Another commenter suggests it is because vmlinuzis a gz file and to extract it, but file vmlinuzoutputs vmlinuz: PE32+ executable (EFI application) Aarch64 (stripped to external PDB), for MS Windows and gzip won't work even with the .gz extension. Did I miss a step?

The Running Linux in a Virtual Machine sample code demonstrates starting a Linux Virtual Machine. But the example only boots to the RAM disk, leaving you in an emergency shell. It does not show how to boot to a disk containing the Linux filesystem. With the sample code unaltered, I can use the Ubuntu RAM disk and kernel files from https://cloud-images.ubuntu.com/releases/23.10/release/unpacked/ to boot Linux like this: ./LinuxVirtualMachine /Users/username/Downloads/ubuntu-23.10-server-cloudimg-arm64-vmlinuz-generic /Users/username/Downloads/ubuntu-23.10-server-cloudimg-arm64-initrd-generic But this fails to fully boot Ubuntu because no root is specified in the bootloader: Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done. Begin: Running /scripts/local-premount ... [ 2.013998] Btrfs loaded, zoned=yes, fsverity=yes Scanning for Btrfs filesystems done. No root device specified. Boot arguments must include a root= parameter. And so Ubuntu drops you to the emergency BusyBox shell. If I mount the root disk image from the release page at https://cloud-images.ubuntu.com/releases/23.10/release/ and specify the root in the bootloader, we get a little further, but Linux can not see the mounted disk: diff --git a/LinuxVirtualMachine/main.swift b/LinuxVirtualMachine/main.swift index bf32924..0977b9e 100644 --- a/LinuxVirtualMachine/main.swift +++ b/LinuxVirtualMachine/main.swift @@ -10,7 +10,7 @@ import Virtualization // MARK: Parse the Command Line -guard CommandLine.argc == 3 else { +guard CommandLine.argc == 4 else { printUsageAndExit() } @@ -25,6 +25,11 @@ configuration.memorySize = 2 * 1024 * 1024 * 1024 // 2 GiB configuration.serialPorts = [ createConsoleConfiguration() ] configuration.bootLoader = createBootLoader(kernelURL: kernelURL, initialRamdiskURL: initialRamdiskURL) +let diskImageURL = URL(fileURLWithPath: CommandLine.arguments[3], isDirectory: false) +let diskImageAttachment = try VZDiskImageStorageDeviceAttachment(url: diskImageURL, readOnly: false) +let storageDeviceConfiguration = VZVirtioBlockDeviceConfiguration(attachment: diskImageAttachment) +configuration.storageDevices = [storageDeviceConfiguration] + do { try configuration.validate() } catch { @@ -71,7 +76,11 @@ func createBootLoader(kernelURL: URL, initialRamdiskURL: URL) -> VZBootLoader { // Use the first virtio console device as system console. "console=hvc0", // Stop in the initial ramdisk before attempting to transition to the root file system. - "rd.break=initqueue" + "rd.break=initqueue", + // Give time for the boot image to be available. + "rootdelay=5", + // Specify the boot image. + "root=/dev/vda" ] bootLoader.commandLine = kernelCommandLineArguments.joined(separator: " ") @@ -104,6 +113,6 @@ func createConsoleConfiguration() -> VZSerialPortConfiguration { } func printUsageAndExit() -> Never { - print("Usage: \(CommandLine.arguments[0]) <kernel-path> <initial-ramdisk-path>") + print("Usage: \(CommandLine.arguments[0]) <kernel-path> <initial-ramdisk-path> <bootable-filesystem-image-path>") exit(EX_USAGE) } Output: ./LinuxVirtualMachine /Users/username/Downloads/ubuntu-23.10-server-cloudimg-arm64-vmlinuz-generic /Users/username/Downloads/ubuntu-23.10-server-cloudimg-arm64-initrd-generic /Users/username/Downloads/ubuntu-23.10-server-cloudimg-arm64.img ...snip... Gave up waiting for root file system device. Common problems: - Boot args (cat /proc/cmdline) - Check rootdelay= (did the system wait long enough?) - Missing modules (cat /proc/modules; ls /dev) ALERT! /dev/vda does not exist. Dropping to a shell! If I instead create a RAW disk image formatted as APFS with the contents of the root drive from the Ubuntu releases page, the mount works but Linux can not read the disk (presumably due to the APFS formatting?): ./LinuxVirtualMachine /Users/username/Downloads/ubuntu-23.10-server-cloudimg-arm64-vmlinuz-generic /Users/username/Downloads/ubuntu-23.10-server-cloudimg-arm64-initrd-generic /Users/username/Desktop/ubuntu-23.10-server.dmg ...snip... Warning: Type of root file system is unknown, so skipping check. mount: mounting /dev/vda on /root failed: Invalid argument done. Begin: Running /scripts/local-bottom ... done. Begin: Running /scripts/init-bottom ... mount: mounting /dev on /root/dev failed: No such file or directory mount: mounting /dev on /root/dev failed: No such file or directory done. mount: mounting /run on /root/run failed: No such file or directory To make that disk image, I did: hdiutil create -size 2g -fs "HFS+" -volname "EmptyImage" ubuntu-23.10-server hdiutil attach ubuntu-23.10-server.dmg diskutil eraseDisk APFS UbuntuFS disk4 // where disk4 is the mounted drive number from the previous step sudo cp -R /path/to/extracted-ubuntu-root-filesystem/* /Volumes/UbuntuFS/ hdiutil detach disk4 What am I missing in order to transition from booting from the RAM disk to booting from the root filesystem?