The Running Linux in a Virtual Machine sample code demonstrates starting a Linux Virtual Machine. But the example only boots to the RAM disk, leaving you in an emergency shell. It does not show how to boot to a disk containing the Linux filesystem. With the sample code unaltered, I can use the Ubuntu RAM disk and kernel files from https://cloud-images.ubuntu.com/releases/23.10/release/unpacked/ to boot Linux like this: ./LinuxVirtualMachine /Users/username/Downloads/ubuntu-23.10-server-cloudimg-arm64-vmlinuz-generic /Users/username/Downloads/ubuntu-23.10-server-cloudimg-arm64-initrd-generic But this fails to fully boot Ubuntu because no root is specified in the bootloader: Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done. Begin: Running /scripts/local-premount ... [ 2.013998] Btrfs loaded, zoned=yes, fsverity=yes Scanning for Btrfs filesystems done. No root device specified. Boot arguments must include a root= parameter. And so Ubuntu drops you to the emergency BusyBox shell. If I mount the root disk image from the release page at https://cloud-images.ubuntu.com/releases/23.10/release/ and specify the root in the bootloader, we get a little further, but Linux can not see the mounted disk: diff --git a/LinuxVirtualMachine/main.swift b/LinuxVirtualMachine/main.swift index bf32924..0977b9e 100644 --- a/LinuxVirtualMachine/main.swift +++ b/LinuxVirtualMachine/main.swift @@ -10,7 +10,7 @@ import Virtualization // MARK: Parse the Command Line -guard CommandLine.argc == 3 else { +guard CommandLine.argc == 4 else { printUsageAndExit() } @@ -25,6 +25,11 @@ configuration.memorySize = 2 * 1024 * 1024 * 1024 // 2 GiB configuration.serialPorts = [ createConsoleConfiguration() ] configuration.bootLoader = createBootLoader(kernelURL: kernelURL, initialRamdiskURL: initialRamdiskURL) +let diskImageURL = URL(fileURLWithPath: CommandLine.arguments[3], isDirectory: false) +let diskImageAttachment = try VZDiskImageStorageDeviceAttachment(url: diskImageURL, readOnly: false) +let storageDeviceConfiguration = VZVirtioBlockDeviceConfiguration(attachment: diskImageAttachment) +configuration.storageDevices = [storageDeviceConfiguration] + do { try configuration.validate() } catch { @@ -71,7 +76,11 @@ func createBootLoader(kernelURL: URL, initialRamdiskURL: URL) -> VZBootLoader { // Use the first virtio console device as system console. "console=hvc0", // Stop in the initial ramdisk before attempting to transition to the root file system. - "rd.break=initqueue" + "rd.break=initqueue", + // Give time for the boot image to be available. + "rootdelay=5", + // Specify the boot image. + "root=/dev/vda" ] bootLoader.commandLine = kernelCommandLineArguments.joined(separator: " ") @@ -104,6 +113,6 @@ func createConsoleConfiguration() -> VZSerialPortConfiguration { } func printUsageAndExit() -> Never { - print("Usage: \(CommandLine.arguments[0]) <kernel-path> <initial-ramdisk-path>") + print("Usage: \(CommandLine.arguments[0]) <kernel-path> <initial-ramdisk-path> <bootable-filesystem-image-path>") exit(EX_USAGE) } Output: ./LinuxVirtualMachine /Users/username/Downloads/ubuntu-23.10-server-cloudimg-arm64-vmlinuz-generic /Users/username/Downloads/ubuntu-23.10-server-cloudimg-arm64-initrd-generic /Users/username/Downloads/ubuntu-23.10-server-cloudimg-arm64.img ...snip... Gave up waiting for root file system device. Common problems: - Boot args (cat /proc/cmdline) - Check rootdelay= (did the system wait long enough?) - Missing modules (cat /proc/modules; ls /dev) ALERT! /dev/vda does not exist. Dropping to a shell! If I instead create a RAW disk image formatted as APFS with the contents of the root drive from the Ubuntu releases page, the mount works but Linux can not read the disk (presumably due to the APFS formatting?): ./LinuxVirtualMachine /Users/username/Downloads/ubuntu-23.10-server-cloudimg-arm64-vmlinuz-generic /Users/username/Downloads/ubuntu-23.10-server-cloudimg-arm64-initrd-generic /Users/username/Desktop/ubuntu-23.10-server.dmg ...snip... Warning: Type of root file system is unknown, so skipping check. mount: mounting /dev/vda on /root failed: Invalid argument done. Begin: Running /scripts/local-bottom ... done. Begin: Running /scripts/init-bottom ... mount: mounting /dev on /root/dev failed: No such file or directory mount: mounting /dev on /root/dev failed: No such file or directory done. mount: mounting /run on /root/run failed: No such file or directory To make that disk image, I did: hdiutil create -size 2g -fs "HFS+" -volname "EmptyImage" ubuntu-23.10-server hdiutil attach ubuntu-23.10-server.dmg diskutil eraseDisk APFS UbuntuFS disk4 // where disk4 is the mounted drive number from the previous step sudo cp -R /path/to/extracted-ubuntu-root-filesystem/* /Volumes/UbuntuFS/ hdiutil detach disk4 What am I missing in order to transition from booting from the RAM disk to booting from the root filesystem?

I have an M3 MacBook Pro, running macOS Sonoma 14.1.1. A tester of our app is running Monterrey, and is experiencing something we can't reproduce, so I need to set up a Monterrey environment. I searched for information here about VMs, and found an article about using VM Frameworks and Xcode, but I am not a programmer in any way. I used to use VMware years ago to host Windows and Linux machines on my Mac, but Google searching leads me to a 2 year old post about Parallels https://www.reddit.com/r/MacOS/comments/tuxfzq/how_can_i_run_macos_monterrey_in_a_virtual/ Can I get a suggestion for how to create and use VMs on Apple Silicon to test older OSs? Thank you.

Hello, I am working with Virtualization framework in an Xcode swift project. I have also installed Parallels Desktop on my mac (Apple Silicon). I would like to convert a Parallels Desktop hard drive (hds file) to an image compatible with my Virtualization framework project. My goal is to run my Parallels Desktop virtual machine in my Xcode project. Is there a way to do that ? Thanks

I created a macOS 14 VM using https://github.com/s-u/macosvm which uses the Virtualization Framework. I want to check if I can use paravirtualized graphics for tensorflow workloads. I followed the steps from https://developer.apple.com/metal/tensorflow-plugin/ but when I run the script from step 4. Verify, I get a segmentation fault (see below). Did anyone try to get this kind of GPU compute in a VM and succeed? /Users/teuf/venv-metal/lib/python3.9/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020 warnings.warn( 2023-11-20 07:41:11.723578: I metal_plugin/src/device/metal_device.cc:1154] Metal device set to: Apple Paravirtual device 2023-11-20 07:41:11.723620: I metal_plugin/src/device/metal_device.cc:296] systemMemory: 10.00 GB 2023-11-20 07:41:11.723626: I metal_plugin/src/device/metal_device.cc:313] maxCacheSize: 0.50 GB 2023-11-20 07:41:11.723700: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:306] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support. 2023-11-20 07:41:11.723968: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:272] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>) zsh: segmentation fault python3 ./tensorflow-test.py Thread 0 Crashed:: Dispatch queue: metal gpu stream 0 MPSCore 0x1999598f8 MPSDevice::GetMPSLibrary_DoNotUse(MPSLibraryInfo const*) + 92 1 MPSCore 0x19995c544 0x199927000 + 218436 2 MPSCore 0x19995c908 0x199927000 + 219400 3 MetalPerformanceShadersGraph 0x1fb696a58 0x1fb583000 + 1129048 4 MetalPerformanceShadersGraph 0x1fb6f0cc8 0x1fb583000 + 1498312 5 MetalPerformanceShadersGraph 0x1fb6ef2dc 0x1fb583000 + 1491676 6 MetalPerformanceShadersGraph 0x1fb717ea0 0x1fb583000 + 1658528 7 MetalPerformanceShadersGraph 0x1fb717ce4 0x1fb583000 + 1658084 8 MetalPerformanceShadersGraph 0x1fb6edaac 0x1fb583000 + 1485484 9 MetalPerformanceShadersGraph 0x1fb7a85e0 0x1fb583000 + 2250208 10 MetalPerformanceShadersGraph 0x1fb7a79f0 0x1fb583000 + 2247152 11 MetalPerformanceShadersGraph 0x1fb6602b4 0x1fb583000 + 905908 12 MetalPerformanceShadersGraph 0x1fb65f7b0 0x1fb583000 + 903088 13 libmetal_plugin.dylib 0x1156dfdcc invocation function for block in metal_plugin::runMPSGraph(MetalStream*, MPSGraph*, NSDictionary*, NSDictionary*) + 164 14 libdispatch.dylib 0x18e79b910 _dispatch_client_callout + 20 15 libdispatch.dylib 0x18e7aacc4 _dispatch_lane_barrier_sync_invoke_and_complete + 56 16 libmetal_plugin.dylib 0x1156dfd14 metal_plugin::runMPSGraph(MetalStream*, MPSGraph*, NSDictionary*, NSDictionary*) + 108 17 libmetal_plugin.dylib 0x115606634 metal_plugin::MPSStatelessRandomUniformOp<float>::ProduceOutput(metal_plugin::OpKernelContext*, metal_plugin::Tensor*) + 876 18 libmetal_plugin.dylib 0x115607620 metal_plugin::MPSStatelessRandomOpBase::Compute(metal_plugin::OpKernelContext*) + 620 19 libmetal_plugin.dylib 0x1156061f8 void metal_plugin::ComputeOpKernel<metal_plugin::MPSStatelessRandomUniformOp<float>>(void*, TF_OpKernelContext*) + 44 20 libtensorflow_framework.2.dylib 0x10b807354 tensorflow::PluggableDevice::Compute(tensorflow::OpKernel*, tensorflow::OpKernelContext*) + 148 21 libtensorflow_framework.2.dylib 0x10b7413e0 tensorflow::(anonymous namespace)::SingleThreadedExecutorImpl::Run(tensorflow::Executor::Args const&) + 2100 22 libtensorflow_framework.2.dylib 0x10b70b820 tensorflow::FunctionLibraryRuntimeImpl::RunSync(tensorflow::FunctionLibraryRuntime::Options, unsigned long long, absl::lts_20230125::Span<tensorflow::Tensor const>, std::__1::vector<tensorflow::Tensor, std::__1::allocator<tensorflow::Tensor>>*) + 420 23 libtensorflow_framework.2.dylib 0x10b715668 tensorflow::ProcessFunctionLibraryRuntime::RunMultiDeviceSync(tensorflow::FunctionLibraryRuntime::Options const&, unsigned long long, std::__1::vector<std::__1::variant<tensorflow::Tensor, tensorflow::TensorShape>, std::__1::allocator<std::__1::variant<tensorflow::Tensor, tensorflow::TensorShape>>>*, std::__1::function<absl::lts_20230125::Status (tensorflow::ProcessFunctionLibraryRuntime::ComponentFunctionData const&, tensorflow::ProcessFunctionLibraryRuntime::InternalArgs*)>) const + 1336 24 libtensorflow_framework.2.dylib 0x10b71a8a4 tensorflow::ProcessFunctionLibraryRuntime::RunSync(tensorflow::FunctionLibraryRuntime::Options const&, unsigned long long, absl::lts_20230125::Span<tensorflow::Tensor const>, std::__1::vector<tensorflow::Tensor, std::__1::allocator<tensorflow::Tensor>>*) const + 848 25 libtensorflow_cc.2.dylib 0x2801b5008 tensorflow::KernelAndDeviceFunc::Run(tensorflow::ScopedStepContainer*, tensorflow::EagerKernelArgs const&, std::__1::vector<std::__1::variant<tensorflow::Tensor, tensorflow::TensorShape>, std::__1::allocator<std::__1::variant<tensorflow::Tensor, tensorflow::TensorShape>>>*, tsl::CancellationManager*, std::__1::optional<tensorflow::EagerFunctionParams> const&, std::__1::optional<tensorflow::ManagedStackTrace> const&, tsl::CoordinationServiceAgent*) + 572 26 libtensorflow_cc.2.dylib 0x28016613c tensorflow::EagerKernelExecute(tensorflow::EagerContext*, absl::lts_20230125::InlinedVector<tensorflow::TensorHandle*, 4ul, std::__1::allocator<tensorflow::TensorHandle*>> const&, std::__1::optional<tensorflow::EagerFunctionParams> const&, tsl::core::RefCountPtr<tensorflow::KernelAndDevice> const&, tensorflow::GraphCollector*, tsl::CancellationManager*, absl::lts_20230125::Span<tensorflow::TensorHandle*>, std::__1::optional<tensorflow::ManagedStackTrace> const&) + 452 27 libtensorflow_cc.2.dylib 0x2801708ec tensorflow::ExecuteNode::Run() + 396 28 libtensorflow_cc.2.dylib 0x2801b0118 tensorflow::EagerExecutor::SyncExecute(tensorflow::EagerNode*) + 244 29 libtensorflow_cc.2.dylib 0x280165ac8 tensorflow::(anonymous namespace)::EagerLocalExecute(tensorflow::EagerOperation*, tensorflow::TensorHandle**, int*) + 2580 30 libtensorflow_cc.2.dylib 0x2801637a8 tensorflow::DoEagerExecute(tensorflow::EagerOperation*, tensorflow::TensorHandle**, int*) + 416 31 libtensorflow_cc.2.dylib 0x2801631e8 tensorflow::EagerOperation::Execute(absl::lts_20230125::Span<tensorflow::AbstractTensorHandle*>, int*) + 132

Hello I have looked at Virtualization framework samples source code provided by Apple. There is something strange: For a linux guest OS, i can see a variable store: let variableStore = try VZEFIVariableStore(creatingVariableStoreAt: xxxx) ... let bootloader = VZEFIBootLoader() ... bootloader.variableStore = variableStore It seems this variable store is linked with NVRAM. For a macOS guest OS, there is no variable store. (The bootloader is created with VZMacOSBootLoader) My question is why macOS guest doesn't need variable store ? Thanks

Hello, I am working on a macOS Virtualization framework project on Xcode. Is there a way to redirect an USB device connected to the host mac, to a virtual machine. I know this is possible with lower layers but i would like to do it with a VZVirtualMachine object. Is it possible ? Thanks

We are currently utilizing VZ with Lima (details: Lima VM and VZ) for our development environment. However, we're encountering a critical issue with the com.apple.Virtualization.VirtualMachine process leading to open file handle exhaustion. When mounting our programming languages dependency cache folder (Which can have a lot of files) into the VZ VM, we encounter an operating system error related to open file limits: /gomodcache/github.com/go-git/go-git/v5@v5.4.2/plumbing/object/patch.go:14:2: open /gomodcache/github.com/go-git/go-git/v5@v5.4.2/plumbing/format/diff/unified_encoder.go: too many open files in system Further investigation revealed an abnormally high number of open files associated with the com.apple.Virtualization.VirtualMachine process. A significant portion of these files are not actively used but remain open. Example Case: A file (/Users/rcurrah/test.txt) created on the Mac host and listed (ls) in the VM remains open even 20 minutes later, as evidenced by the following command output: ❯ lsof | grep 11208 | grep test.txt COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME com.apple 11208 rcurrah 4823r REG 1,13 0 46200882 /Users/rcurrah/test.txt Steps to Reproduce the Issue: To reproduce the file handle exhaustion follow the below steps. This process will create a large number of files on the Mac host, listing them on the VZ VM, and then verifying their open status using lsof. Setup the VZ Environment with Sharing: Create a VZ VM with your home directory shared to the VM. Create a Test Directory on the Mac Host: Create a new directory on your Mac host, e.g., mkdir ~/test-file-exhaustion. Generate a Large Number of Files: Navigate to the created directory: cd ~/test-file-exhaustion. Use a loop to create a large number of files, e.g., for i in {1..10000}; do touch "file_${i}.txt"; done. This will create 10,000 files named file_1.txt, file_2.txt, etc. List Files in the VM: Access the VZ VM shell. Navigate to the mounted directory and list the files using the ls command, e.g., ls /path/to/mounted/test-file-exhaustion. Check Open Files on Mac Host: Exit the VM and return to your Mac host terminal. Use the lsof command to check for open files related to the com.apple.Virtualization.VirtualMachine process: lsof | grep "$(pgrep com.apple.Virtualization.VirtualMachine)" | grep 'test-file-exhaustion' | wc -l. Document the Output: Record the output of the lsof command. Note the number of open files. Verify File Closure (or Lack Thereof): After a certain period, e.g., 20 minutes, repeat the lsof command to see if the files are still open, indicating that they haven’t been closed properly by the process. Given these observations, we have a couple of questions: Is this behavior of com.apple.Virtualization.VirtualMachine retaining open file handles a known issue or a bug? Should VZ be managing the closure of these file handles more efficiently, especially when they are no longer in use? This issue is impacting our development workflow significantly. Any guidance or insights on resolving this would be highly appreciated. Thank you for your attention to this matter. Best regards, Ryan

Filed as rdar://FB11975037 When macOS Ventura is run as a guest OS within the virtualization framework, the main menu bar items will not be displayed correctly if VZMacGraphicsDisplayConfiguration defines a large resolution. The menu bar titles appear to be using the same color as the menu bar itself. When the Appearance is set to Light, the menu bar items are effectively invisible. When the Appearance is set to Dark, the menu bar items are drawn in what looks like a disabled state. This only affects the menu bar item titles on the left-hand side. The date-time and menu bar icons on the right side are always displayed in the correct color. This appears to be a regression in macOS Ventura as this issue is not present in macOS 12 running as a guest. This bug can be easily reproduced using Apple's own Virtualization sample code titled: "Running macOS in a Virtual Machine on Apple Silicon Macs" Steps to reproduce: Follow the sample code instructions for building and installing a VM.bundle. Before running 'macOSVirtualMachineSampleApp', change the VZMacGraphicsDisplayConfiguration to use: width = 5120, height = 2880, ppi = 144. Run 'macOSVirtualMachineSampleApp' and notice that the menu bar titles on the left side of the screen are not correctly drawn in the guest instance. This has been tested on: Host: macOS 13.1 Guest: macOS 13.x (All versions) Hardware: MBP 14" M1 Pro 32GB/2TB Is there anything that can be done to resolve this issue?

Logging in with my Apple ID anywhere in the system (feedback assistant, Xcode, iCloud, etc.) fails when running under virtualization. Is this a known 'issue'? (networking in general is working fine)