nbd-vram: Using NVIDIA GPU VRAM as Linux Swap Space

Using VRAM as Swap Space to Extend System Memory

nbd-vram is a utility designed for Linux users—particularly those with laptops featuring soldered RAM—to utilize unused NVIDIA GPU VRAM as a swap device. By treating VRAM as a block device, the tool allows the system to spill over into GPU memory before resorting to slower SSD-based swap, effectively increasing the addressable memory of the system.

In a typical configuration, the memory overflow order is designed as follows:

  1. Physical RAM: Primary system memory fills first.
  2. VRAM (via nbd-vram): Fast spillover via PCIe.
  3. zram: Compressed RAM managed by the CPU.
  4. SSD Swap: The final resort when all other tiers are exhausted.

Technical Architecture and Implementation

nbd-vram avoids the need for custom kernel modules by utilizing the Network Block Device (NBD) protocol. The implementation follows this data path:

Kernel swap subsystem $\rightarrow$ /dev/nbdX $\rightarrow$ nbd kernel driver $\rightarrow$ Unix socket $\rightarrow$ nbd-vram daemon $\rightarrow$ cuMemcpyHtoD/DtoH $\rightarrow$ GPU VRAM

Why NBD instead of P2P API?

The developer chose the NBD approach to bypass restrictions in NVIDIA's consumer-grade drivers. While the nvidia_p2p_get_pages_persistent API allows pinning VRAM pages for direct CPU access via ioremap_wc, NVIDIA gates this functionality to Quadro and datacenter SKUs. On GeForce GPUs, this API returns EINVAL. Additionally, attempting to directly ioremap_wc the BAR1 physical address fails because consumer GPUs typically only map ~16 MiB of BAR1 (the display framebuffer), returning zeros for the rest of the address space.

By using cuMemcpyHtoD and cuMemcpyDtoH, nbd-vram can operate on any CUDA-supported GPU without requiring special permissions or enterprise hardware.

Performance and Configuration

Benchmarks

On an RTX 3070 Laptop, nbd-vram achieved a sequential throughput of approximately 1.3 GB/s using 4M blocks. The developer notes that latency is lower than NVMe because the data path travels over PCIe directly to the GPU rather than through a storage controller.

Setup and Requirements

To run nbd-vram, the system requires:

  • An NVIDIA GPU with CUDA support.
  • NVIDIA drivers providing libcuda.so.1 (the full CUDA toolkit is not required).
  • Linux kernel 3.0+ with the nbd module.
  • The nbd-client package, gcc, and make.

Users can configure the amount of VRAM to allocate via the VRAM_SETUP_SIZE_MB environment variable in the systemd service file. The daemon is designed to be flexible; if the requested size is unavailable, it backs off in 512 MiB increments until it can successfully allocate memory.

Power and Resource Management

To prevent excessive battery drain on laptops, nbd-vram includes power-aware management. When enabled, the service automatically stops when the device is unplugged from AC power (or hits a specific battery threshold) and restarts upon reconnection.

Community Insights and Critical Analysis

While the project provides a creative solution for memory-constrained hardware, the community has raised several technical concerns regarding its efficiency and stability:

Throughput and Overhead

Some users pointed out that the reported 1.3 GB/s throughput is significantly lower than the theoretical limits of PCIe 4.0 x16 (64 GB/s) and GDDR6 memory (448 GB/s).

"This RTX 3070 chip is on PCIe 4.0 x16 which should give 64GB/s... Swapping to an NVMe drive would be twice as fast, but with higher latency."

This suggests that the NBD-over-Unix-socket architecture introduces significant overhead compared to a native kernel-level block driver.

VRAM Pressure and Stability

There are concerns regarding how the system handles VRAM contention. Since the VRAM is allocated for swap, other GPU-intensive applications (like Wayland compositors or LLMs) may face memory pressure, potentially leading to system crashes.

"Running low on VRAM can easily crash the whole desktop... I just had a few of such crashes with Hyprland+llama-server+KVM."

Alternative Implementations

Discussion highlighted other similar attempts to use VRAM as storage, including vramblk (using OpenCL) and GpuRamDrive (for Windows), noting that most such implementations suffer from a performance drop that makes them feel like "traditional swapping" rather than a high-performance memory extension.

Sources