virtio-nvgpu enables near‑native NVIDIA GPU performance in KVM guests
Near‑native NVIDIA GPU access inside a KVM guest
Takeaway: virtio-nvgpu forwards NVIDIA kernel‑driver ioctls between host and guest at the driver ABI level, allowing a Linux KVM guest to run the unmodified NVIDIA user‑mode driver and achieve rendering performance within 2 % of bare‑metal, with essentially no extra CPU cost.
How it works
- Guest kernel driver (GPL) registers the usual
/dev/nvidia*nodes. On eachioctl()it serializes the raw request onto a virtqueue; onmmap()it maps a shared memory window directly into the guest process. - Device crate (Apache‑2.0, Rust) runs in the VMM, receives those virtqueue messages, translates embedded file descriptors and pointers, and forwards the ioctl to the host's NVIDIA driver. Buffer bookkeeping lives here.
- Event virtqueue notifies the guest when the GPU has completed work, eliminating busy‑polling.
- Isolation plan – a future per‑guest sandboxed helper (the isolate component) will hold the real device file descriptors; today the VMM process holds them directly.
The architecture mirrors chromeos/virtio-media: a GPL guest driver sits beside a permissively‑licensed, VMM‑agnostic device crate, with all VMM interactions expressed as traits.
Performance results (RTX 3060, driver 595.99.02)
| Metric | Guest (virtio‑nvgpu) | Bare‑metal host |
|---|---|---|
| Frame time for a typical game draw (≥2 ms) | ‑0.4 % to +1.7 % (within noise) | — |
| Frame time for ultra‑light draws (≤0.5 ms) | Up to +40.8 % (waiting cost dominates) | — |
| CPU usage for a 100 fps un‑paced run (12 s) | 0.37 s | 0.40 s |
| Host‑guest crossings per frame | ≈0.02 (one crossing per ~59 frames) | Thousands (Venus) |
| Multi‑guest scaling (4 guests on one RTX 3060) | 103.7 fps total, each ~25.8 fps, identical to single‑guest performance | — |
Interpretation: For GPU‑bound workloads the guest is indistinguishable from bare metal; the only measurable penalty appears when the frame budget is smaller than the GPU wake latency (~0.02 ms).
---\n
Supported use cases
- Vulkan and OpenGL rendering (headless EGL works)
- Wayland compositor inside the guest, presenting off‑screen frames
- CUDA memory allocation and zero‑copy interop with Vulkan/GL
- NVENC encoding directly from guest‑side CUDA buffers (H.264/H.265 output)
The design deliberately excludes physical scan‑out, MIG/SR‑IOV, and full unified virtual memory (cudaMallocManaged).
Driver ABI handling
NVIDIA’s kernel driver ABI changes between releases. virtio-nvgpu ships explicit ABI profiles:
| Profile | Covers |
|---|---|
535.129.03 |
535.129.03 up to next profile |
580.178.04 |
580.178.04 up to next profile |
595.71.05 |
595.71.05 and newer |
Any driver older than 535.129.03 is rejected. Newer drivers are accepted under the last profile until a new profile is added. The profiles are generated mechanically from NVIDIA’s open‑gpu‑kernel‑modules sources, not hand‑typed.
Comparison with other GPU virtualization approaches
| Approach | Guest driver | API translation | GPU sharing | NVENC support | Isolation |
|---|---|---|---|---|---|
| VFIO passthrough | Host driver bound to guest PCI device | None (native) | One VM per GPU (or dedicated card) | Works (native) | Strong (IOMMU isolates guest memory) |
| virtio‑gpu + Venus | Guest Mesa driver (virgl) | Per‑draw‑call serialization & replay | Multiple VMs share GPU | Not viable (buffers live on host) | Moderate |
| virtio‑nvgpu (this project) | Unmodified NVIDIA user‑mode driver | Forward ioctls at driver ABI level (≈ 1 crossing per frame) | Multiple VMs share a single card (tested up to 4) | Works, zero‑copy | Weak – all ioctls are forwarded; host driver is in the TCB |
Key insight: By moving the translation boundary from the graphics API to the kernel driver,
virtio‑nvgpueliminates the massive per‑draw‑call overhead of Venus while still allowing multiple guests to share a GPU.
Security and isolation considerations
- No IOMMU boundary separates guest GPU work from host memory; the host NVIDIA driver runs in the host’s IOMMU domain and is effectively part of the trusted computing base.
- The current implementation forwards all driver ioctls (except those explicitly refused by the ABI profile). A permissive flag (
--permissive-abi) can relax this further for debugging. - Future work (
isolate/) aims to sandbox the real device file descriptors in a per‑guest helper process, but is not yet implemented. - Compared to VFIO, the attack surface is larger: a compromised guest could issue any allowed ioctl, similar to running multiple processes on the host that share the GPU.
Community feedback (Hacker News highlights)
- @refibrillator notes the three main GPU‑sharing methods (VFIO, virtio‑gpu/Venus, virtio‑nvgpu) and points out that
virtio‑nvgpuoffers the weakest isolation because the host driver is in the TCB. - @orphereus asks why not just use normal GPU passthrough; the answer is that passthrough cannot share a GPU across VMs.
- @markasoftware asks how this differs from gVisor’s
nvproxy; the README listsnvproxyas the direct inspiration but does not detail the differences. - @ericd expresses excitement about a performant gaming VM without VFIO complexity.
- @majorchord inquires about Windows guest support; the current project targets Linux guests only.
Repository layout
| Directory | License | Purpose |
|---|---|---|
driver/ |
GPL‑2.0 | Guest kernel module that forwards ioctls and mmaps via virtqueues |
device/ |
Apache‑2.0 | VMM‑agnostic Rust crate implementing the virtio device and ABI translation |
isolate/ |
Apache‑2.0 | Design notes for a future per‑guest sandboxed helper (no code yet) |
gen/ |
— | Auto‑generated ABI tables derived from NVIDIA sources |
protocol/ |
BSD‑3‑Clause OR GPL‑2.0+ | Shared wire format and ABI definitions |
Getting started
- Build the guest driver (
driver/) and load it inside the KVM guest. - Integrate the
device/crate into your VMM by implementing the required traits (descriptor chain I/O, event handling, guest/host memory mapping). - Run a Vulkan or Wayland application inside the guest; the README’s benchmark scripts (
BENCHMARKS.md) verify near‑native performance.
Limitations and future work
- Tested with up to four guests on a single RTX 3060; more guests are untested.
- Only driver versions 595.99.02 (RTX 3060) and 615.71.09 (RTX A2000) have been exercised; other cards are untested.
- CUDA functionality is forwarded but not thoroughly validated beyond enumeration.
- No Windows guest support yet; the project targets Linux only.
- Isolation improvements (per‑guest sandbox, seccomp filters) are planned but not present.
License
The repository contains three licensing zones: GPL‑2.0 for the guest driver, Apache‑2.0 for the device crate and isolate design, and BSD‑3‑Clause (or GPL‑2.0+) for the shared protocol headers.
In summary, virtio-nvgpu demonstrates that forwarding NVIDIA driver ioctls at the ABI level can give KVM guests near‑bare‑metal GPU performance while allowing multiple VMs to share a single card, albeit with weaker isolation than VFIO passthrough.