virtio‑nvgpu – Near‑native NVIDIA GPU access for KVM guests

What it is

• A pair of components that let a Linux KVM guest use an NVIDIA GPU through the driver’s ioctl ABI, not by translating graphics APIs.
• The guest runs NVIDIA’s unmodified user‑mode drivers (Vulkan, OpenGL, CUDA, NVENC) and talks directly to the host’s kernel driver via a virtio device.

Why it matters

• Existing solutions either:
  • VFIO passthrough – gives the whole GPU to one VM, wasting resources in multi‑tenant setups.
  • API‑level proxies (e.g., Venus) – serialize every draw call, adding latency and CPU overhead, and they cannot expose the GPU’s encoder to the guest.
• By forwarding driver‑level ioctls, virtio‑nvgpu keeps the guest’s command‑buffer generation local, so the only VM exits are the few ioctl submissions. This yields GPU performance within 2 % of bare‑metal for typical frame times (>2 ms) and no extra CPU cost.

How it works

1. Guest kernel driver (driver/, GPL‑2.0) registers the usual /dev/nvidia* nodes. An ioctl or mmap from a user program is packaged onto a virtqueue and sent to the host.
2. Device crate (device/, Apache‑2.0, written in Rust) implements the virtio device. It receives the requests, rewrites any embedded pointers or file descriptors, and forwards the ioctl to the host’s NVIDIA driver. It also handles the reverse‑direction virtqueue that notifies the guest when the GPU has completed work.
3. Protocol (protocol/) defines the wire format and ABI tables (generated in gen/).
4. Isolate (isolate/) is a design note for a future per‑guest sandboxed helper that will hold the real device file descriptors; today the VMM process itself performs the forwarding.

Key capabilities

• Vulkan & OpenGL rendering (headless, Wayland compositor inside the VM)
• CUDA memory allocation and zero‑copy interop with Vulkan/OpenGL
• NVENC encoding directly from guest‑side buffers (H.264/H.265)
• Multiple guests on a single GPU (tested with four RTX 3060 guests sharing the card evenly)
• Driver‑ABI versioning – explicit support for NVIDIA driver ranges 535‑595 (profiles in gen/). Older drivers are rejected, newer ones fall back to the latest profile.

Performance highlights (RTX 3060, driver 595.99.02)

Metric	Result
Frame time overhead	‑0.4 % to +7 % depending on frame length; ≤2 % for frames ≥2 ms
CPU usage	Guest uses 0.37 s of CPU for a 12 s, ~100 fps run vs. host’s 0.40 s (essentially identical)
VM‑crossings	≈0.02 crossings per frame (≈1 crossing per 59 frames)
Multi‑guest scaling	4 guests → ~103 fps total, same as a single guest’s 102.9 fps
NVENC	Works with zero‑copy, same bitrate as bare‑metal

Maturity

• Core path (ioctl forwarding, mmap sharing) is functional and benchmarked.
• Tested with real games and synthetic loads on two cards (RTX 3060, RTX A2000).
• Limitations: only up to four guests have been exercised; no full isolation (the host driver runs in the TCB); some ioctl classes are still unfiltered; CUDA beyond enumeration is untested.

Typical use case A headless streaming server that runs a game or GPU‑intensive application inside a VM, composites the scene, encodes it with NVENC, and streams the compressed video to remote clients. The VM sees a real NVIDIA driver, so existing game binaries and CUDA‑based pipelines run unchanged, while the host retains exclusive control of the physical monitor.

How to adopt

• Implement the small set of traits defined in device/ (descriptor‑chain read/write, event queue, guest/host memory mapping) in your VMM.
• Load the GPL guest driver inside the VM.
• Configure the VMM to expose the virtio‑nvgpu device to the guest.
• Ensure the host runs a supported NVIDIA driver version.

Licensing

• Guest driver: GPL‑2.0 (needs kernel symbols).
• Device crate & shared protocol: Apache‑2.0 (permissive, VMM‑agnostic).
• Protocol headers are dual‑licensed BSD‑3‑Clause OR GPL‑2.0+ so both sides can include them.

Related projects

• gVisor nvproxy – container‑level NVIDIA ioctl forwarding; the ABI handling in virtio‑nvgpu is derived from it.
• ChromeOS virtio-media – repository layout inspiration.
• WSL2 /dev/dxg – production driver‑level GPU proxy for Windows, showing the general viability of the approach.

Where to learn more

• BENCHMARKS.md – detailed measurement methodology and raw data.
• ARCHITECTURE.md – deeper dive into the memory‑sharing and event model.
• protocol/ – the wire format and generated ABI tables.

---

virtio‑nvgpu provides a practical path to near‑bare‑metal NVIDIA GPU performance inside KVM guests, enabling multi‑tenant GPU‑accelerated workloads such as game streaming, AI inference, or any CUDA‑based application without sacrificing host resources.