cuTile Rust 0.2.0 Enables Safe, Data-Race-Free GPU Kernels in Rust

cuTile Rust 0.2.0 makes GPU kernel programming safe and fast in Rust

cuTile Rust 0.2.0 introduces a tile‑based programming model that carries Rust's ownership guarantees across the GPU launch boundary, allowing developers to write memory‑safe, data‑race‑free kernels without sacrificing performance.

Core Idea: Ownership Discipline Extends to the GPU

  • Safety first – Mutable tensors are partitioned into disjoint tiles before launch, guaranteeing exclusive &mut access for each tile. Immutable tensors are shared as & references.
  • Zero‑runtime overhead – The compiler maps single‑threaded tile code to CUDA thread blocks and manages shared memory automatically. Benchmarks on an NVIDIA B200 show safe GEMM within 0.3 % of a hand‑written Tile IR variant and element‑wise throughput of ~7 TB/s.
  • Unified API – The same model supports synchronous launches, asynchronous pipelines, and CUDA graph replay, all via a high‑level host‑side API.

How It Works: Macro‑Driven Kernel Generation

The #[cutile::module] macro captures the Rust AST of each kernel, embeds it in the host binary, and JIT‑compiles it to CUDA Tile IR at runtime.

use cutile::prelude::*;

#[cutile::module]
mod kernel {
    use cutile::core::*;

    #[cutile::entry()]
    fn add<const B: i32>(
        z: &mut Tensor<f32, { [B] }>,
        x: &Tensor<f32, { [-1] }>,
        y: &Tensor<f32, { [-1] }>,
    ) {
        let tx = load_tile_like(x, z);
        let ty = load_tile_like(y, z);
        z.store(tx + ty);
    }
}

fn main() -> Result<(), Error> {
    let x = api::ones::<f32>(&[1024]);
    let y = api::ones::<f32>(&[1024]);
    let z = api::zeros::<f32>(&[1024]).partition([128]);

    let (_z, _x, _y) = kernel::add(z, x, y).sync()?;
    Ok(())
}
  • The macro transforms add into a GPU kernel and generates a host‑side launcher.
  • The launch grid (8, 1, 1) is inferred from the partition (1024 / 128 = 8 tiles).
  • The kernel signature enforces the access discipline: exclusive mutable output (z) and shared read‑only inputs (x, y).

Performance Highlights from the Paper

  • Element‑wise ops – 7 TB/s on NVIDIA B200 (≈ 91 % of peak memory bandwidth).
  • Dense GEMM (f16) – 2 PFlop/s on B200 (≈ 92 % of peak), within 0.3 % of a low‑level Tile IR implementation.
  • LLM inference (Grout) – 171 tokens/s for Qwen3‑4B on RTX 5090 and 82 tokens/s for Qwen3‑32B on B200, matching state‑of‑the‑art memory‑bound performance.

"The safety overhead is effectively free; safe Rust persistent GEMM reaches 2.07 PFlop/s at M=N=K=8192, within 0.3 % of the corresponding low‑level Tile IR variant." – authors, arXiv:2606.15991

Getting Started

Prerequisites

  • NVIDIA GPU with compute capability sm_80 or higher (Ampere+). CUDA 13.3 is recommended.
  • Rust 1.89+ on Linux (tested on Ubuntu 24.04).

Installation Steps

# Install Rust (stable)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup default stable

# Install CUDA 13.3 (follow NVIDIA's instructions)
# Set environment variable
export CUDA_TOOLKIT_PATH=/usr/local/cuda-13

Add the path to .cargo/config.toml if desired:

[env]
CUDA_TOOLKIT_PATH = { value = "/usr/local/cuda-13", relative = false }

Verify the Installation

cargo run -p cutile-examples --example hello_world

You should see: Hello, I am tile <0, 0, 0> in a kernel with <1, 1, 1> tiles.

Project Structure

cutile                 # User‑facing crate
├─ cutile-macro       # Procedural macro implementation
├─ cutile-compiler    # Compiler pipeline to Tile IR
├─ cuda-async         # Async CUDA execution utilities
└─ cuda-core          # Safe CUDA API bindings

cutile-kernels         # Reusable kernel library
cutile-ir              # Pure Rust Tile IR builder
cuda-bindings          # Low‑level NVIDIA bindings (NVIDIA license)

All crates except cuda-bindings are Apache‑2.0 licensed.

How cuTile Rust Differs from CUDA‑Oxide

"How does this compare to NVIDIA's CUDA‑oxide? The latter is similar in syntax to CUDARC on the host side, but replaces the normal‑CUDA‑kernel (in C++‑ish) on device side with Rust." – HN comment

  • Safety model – cuTile Rust enforces data‑race freedom through tile partitioning, while CUDA‑oxide provides a more direct mapping of Rust code to PTX without the tile abstraction.
  • Target language – cuTile Rust compiles to CUDA Tile IR, a higher‑level intermediate representation that enables aggressive optimizations (e.g., FP4 packing) and portability across NVIDIA architectures. CUDA‑oxide emits raw PTX.
  • Kernel semantics – cuTile kernels are written with single‑threaded semantics; the compiler expands them to thread blocks. CUDA‑oxide expects the programmer to manage SIMT details manually.
  • Ecosystem – cuTile Rust ships with a full host‑side tensor API, async support, and integration with projects like Hugging Face's Grout. CUDA‑oxide is primarily a research compiler.

Community and Future Directions

  • The project is in an early research stage; API breakage and missing features are expected.
  • Contributions are welcomed via the CONTRIBUTING.md guide.
  • Planned enhancements include broader low‑precision support, richer tensor operations, and tighter integration with existing Rust ML stacks (e.g., Burn, Candle).

References

  • Paper: Fearless Concurrency on the GPU (arXiv:2606.15991) – detailed methodology and benchmarks.
  • GitHub: https://github.com/NVlabs/cutile-rs – source code, examples, and Nix flake for development.
  • Related projects:
    • Hugging Face Grout – LLM inference engine built on cuTile Rust.
    • cuTile Python – Python bindings for the same Tile IR backend.
    • CUDA‑oxide – Experimental Rust‑to‑CUDA compiler from NVlabs.

Sources