NVIDIA gds‑nvidia‑fs

What it is – A Linux kernel module (nvidia-fs.ko) that implements NVIDIA’s GPUDirect Storage (GDS) driver. It lets supported storage systems move data directly into or out of GPU memory via DMA/RDMA, bypassing the CPU and host memory.

Why it matters – Modern AI/ML workloads often need to feed terabytes of data to GPUs at high speed. Traditional paths copy data through the CPU, creating a bandwidth bottleneck. This driver enables a zero‑copy path, which can dramatically increase throughput and reduce latency for training, inference, and data‑intensive analytics.

---

Core capabilities

Capability	Details
Direct GPU I/O	Moves data between NVMe/NVMe‑OF, ScaleFlux computational storage, or RDMA‑enabled filesystems and GPU memory without staging in host RAM.
Supported storage back‑ends	• XFS & EXT4 (ordered mode) on NVMe/NVMe‑OF/ScaleFlux CSD • NFS over RDMA (MOFED ≥ 5.1) • Distributed RDMA filesystems: DDN Exascaler, WekaFS, VAST
Hardware requirements	NVIDIA Tesla or Quadro GPUs (Pascal‑generation or newer), compatible NVMe or distributed storage, Linux kernel ≥ 4.15, MOFED ≥ 5.1, CUDA ≥ 10.0, driver ≥ 418.40
Kernel integration	Built as a loadable kernel module (nvidia-fs.ko).
Documentation	Official GDS docs: https://docs.nvidia.com/gpudirect-storage/index.html

---

Typical use‑case workflow

1. Prepare the environment – Install a supported NVIDIA GPU, appropriate storage (e.g., NVMe SSDs or an RDMA‑enabled filesystem), and the required software stack (Linux kernel, MOFED, CUDA toolkit, NVIDIA driver).
2. Build the driver

   cd src
   export CONFIG_MOFED_VERSION=$(ofed_info -s | cut -d '-' -f 2)
   sudo make
   

3. Load the module

   sudo insmod nvidia-fs.ko
   

4. Use the cufile APIs from your CUDA application (or from libraries such as RAPIDS, PyTorch‑Dataloader, TensorFlow) to open files on the supported storage and read/write directly into GPU buffers.
5. Benefit – Higher effective I/O bandwidth and lower CPU overhead, which translates into faster data‑pipeline throughput for large‑scale model training or inference.

---

Getting started

# Clone the repo (example URL)
git clone https://github.com/NVIDIA/gds-nvidia-fs.git
cd gds-nvidia-fs/src

# Export the MOFED version detected on the host
export CONFIG_MOFED_VERSION=$(ofed_info -s | cut -d '-' -f 2)

# Build and install the kernel module
sudo make
sudo insmod nvidia-fs.ko

After loading, the driver registers with the kernel and becomes usable by applications that call the NVIDIA GPUDirect Storage (cufile) library.

---

Who should use this

• AI/ML engineers building high‑performance data pipelines for training large models.
• Data‑center operators who want to maximize GPU utilization by reducing CPU‑side I/O bottlenecks.
• Researchers working on storage‑aware GPU computing, computational storage, or RDMA‑based systems.

---

Limitations & notes

• The driver only works on Linux kernels 4.15+ and requires the MOFED (Mellanox OFED) stack for RDMA support.
• Filesystems must be mounted in ordered mode for XFS/EXT4; other mount options are not supported.
• It is a low‑level kernel component; users typically interact with it via higher‑level libraries (e.g., NVIDIA’s cufile API) rather than directly.

---

In short – nvidia-fs is the kernel‑level piece that makes NVIDIA’s GPUDirect Storage work, enabling ultra‑fast, zero‑copy data movement from modern storage systems straight into GPU memory, a key building block for next‑generation AI workloads.