Apple Silicon macOS VMs: Accelerating LLM Inference with Metal Capability Shim

LLM Inference in macOS VMs can be 11-16x faster by unlocking Metal capabilities

Running Large Language Models (LLMs) via llama.cpp inside a macOS guest using Apple's Virtualization.framework is significantly slower than bare-metal performance because the virtual GPU reports a conservative capability profile. By using a process-scoped compatibility shim to report newer Metal capabilities, inference speeds for TinyLlama 1.1B improved by 11.08x for prompt processing and 16.36x for token generation on an M1 Ultra.

This performance gain occurs because the shim allows llama.cpp to select modern Metal kernels—such as SIMD-group matrix and bfloat16 paths—that the host hardware supports but the virtual device normally hides.

The Root Cause: Paravirtualized GPU Capability Caps

Apple's Virtualization.framework uses paravirtualization, where a virtual graphics device in the guest submits work to a purpose-built driver that the host executes on the physical GPU. Unlike PCI passthrough (VFIO) used in some Linux environments, the host maintains strict control over the hardware.

In stock macOS VMs, the virtual device reports a capability profile roughly equivalent to an Apple 5-era family with only 32 KB of maximum threadgroup memory. Because modern Metal applications query the device at runtime to select the most efficient kernels, llama.cpp defaults to slower, legacy paths based on these conservative reports, even though the underlying physical Apple Silicon GPU is capable of much more.

The Solution: Process-Scoped Metal Capability Shim

To bypass these restrictions, Cua developed a Metal capability shim—a compatibility layer inserted between the application and the API. This shim intercepts Metal capability queries for a specific guest process and modifies the returned values to unlock higher-performance paths.

Key Capability Changes

Capability Stock Guest Unlocked Profile
supportsFamily:1009 False True
SIMD-group matrix Off On
SIMD-group reduction Off On
bfloat16 Off On
Max threadgroup memory 32 KB 64 KB

The shim specifically answers supportsFamily: through Apple family 9 (1009) and doubles the reported maximum threadgroup memory to 64 KB. This is sufficient to trigger the selection of newer SIMD-group and bfloat16 paths in llama.cpp without requiring changes to the guest kernel or physical GPU assignment.

Performance Benchmarks

Tests were conducted on an Apple M1 Ultra (48-core GPU) running macOS 26.6.1, with a guest Tahoe image (macOS 26.5.2) in Lume 0.5.1.

TinyLlama 1.1B (Q4_K_M)

Workload Bare-metal Host Stock Guest Unlocked Guest Guest Speedup
Prompt processing (512 tok) 4,871.99 tok/s 431.86 tok/s 4,786.70 tok/s 11.08x
Token generation (128 tok) 286.71 tok/s 12.63 tok/s 206.60 tok/s 16.36x

Prompt processing in the unlocked VM reached 98.25% of bare-metal speed.

Gemma 4 12B (QAT Q4_0)

Workload Bare-metal Host Stock Guest Unlocked Guest Guest Speedup
Prompt processing (512 tok) 517.88 tok/s 71.66 tok/s 515.76 tok/s 7.20x
Token generation (128 tok) 52.38 tok/s 3.41 tok/s 49.67 tok/s 14.54x

MLX-LM Compatibility

Testing with MLX-LM 0.31.3 and Llama-3.2-3B-Instruct-4bit showed flat performance (approx. 1.0x ratio). This indicates that MLX-LM was already utilizing efficient paths in the stock VM, helping the developers refine the shim to avoid requesting residency sets that the paravirtualized device cannot support.

Deployment and Implementation

The shim is released as a research artifact in the libs/lume/metal-capability-shim directory of the Cua repository. Implementation requires the following steps:

  1. Build: Compile the architecture-specific dylibs using the provided build scripts.
  2. Host Configuration: Enable the unrestricted feature level via the host terminal: defaults write com.apple.gpusw.ParavirtualizedGraphics ForceUnrestrictedDeviceFeatureLevel -bool true
  3. Injection: Launch the workload in the guest using DYLD_INSERT_LIBRARIES to scope the shim to that specific process.

Limitations and Technical Constraints

  • Version Sensitivity: The shim relies on private guest Metal implementation details and may break between macOS releases.
  • Process Scope: It only affects the injected process and its children; hardened binaries may reject the library injection.
  • Virtualization Overhead: While it unlocks GPU kernels, it does not eliminate the inherent overhead of the Virtualization.framework bridge.
  • Narrow Validation: Results are currently validated for M1 Ultra and Tahoe guests; other Apple Silicon generations require independent testing.

Community Perspective

Discussion among technical users emphasizes that this is not a general llama.cpp speedup for all Mac users, but specifically a fix for those running inference within Virtualization.framework VMs. Some users questioned why Apple restricts the Metal profile in VMs, noting that the hardware is highly capable but the software abstraction layer is conservative.

Sources

Related