luminal-ai/luminal

Inference at the speed of light.

What it solves

Luminal is designed to overcome the performance bottlenecks and complexity explosion found in traditional deep learning frameworks. It replaces eager-first execution with a high-performance, general-purpose inference compiler that optimizes computation graphs ahead-of-time, reducing runtime overhead and maximizing hardware utilization.

How it works

Luminal treats all neural networks as static computation graphs. It reduces complex operations into 15 primitive ops (unary, binary, and other) to maintain a minimal core. Instead of relying on hand-written heuristics for optimization, it uses a search-based approach to discover the most efficient execution paths—automatically discovering optimizations like Flash Attention. It is written in Rust and interacts directly with accelerator APIs (CUDA, Metal) to avoid abstraction layers.

Who it’s for

It is intended for developers and researchers who need maximum inference speed on any device, as well as PyTorch users who want to use Luminal as a compiler backend via torch.compile.

Highlights

  • High Performance: Capable of running Q8 Llama 3 8B at ~80% of theoretical max performance on an H100.
  • PyTorch Integration: Works as a compiler backend for PyTorch models.
  • Minimalist Architecture: Uses a RISC-style approach with only 15 primitive operations.
  • Symbolic Dimensions: Natively supports dynamic shapes through symbolic dimensions for aggressive specialization.
  • Native Rust Implementation: Statically-linked Rust crate that interacts directly with hardware APIs.

Related

  • Project
    EricLBuehler/mistral.rs

    mistral.rs is a Rust‑based engine for fast, flexible local LLM inference. It supports many model formats (Hugging Face, GGUF), automatic hardware‑aware quantization, multimodal inputs, and built‑in agentic features (tool calling, Python/shell execution, web search). It can serve OpenAI‑compatible and Anthropic‑compatible APIs, offers a web UI, and provides Rust and Python SDKs for embedding.

  • Project
    lucasjinreal/Crane

    Crane is a Rust‑based inference engine built on the Candle ML library. It loads GGUF models (LLMs, vision‑language, OCR, ASR, TTS, etc.) and runs them fast on CPU, CUDA, Apple Metal, or AMD ROCm, providing an OpenAI‑compatible server and easy‑to‑use Rust APIs.

  • Project
    facebookincubator/AITemplate

    A Python framework that transforms deep neural networks into optimized CUDA/HIP C++ code for high-performance inference serving on NVIDIA and AMD GPUs.

  • Project
    huggingface/candle

    A minimalist ML framework for Rust that enables lightweight, Python-free serverless inference and browser-based model execution.

  • Project
    NVIDIA/TensorRT-LLM

    TensorRT‑LLM is NVIDIA’s open‑source library that compiles large language (and visual‑generation) models into ultra‑fast TensorRT inference engines. It provides custom CUDA kernels, quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token throughput while keeping accuracy. The Python‑friendly runtime integrates with Triton, Kubernetes, and Jetson, and includes docs and examples for models like GPT‑OSS‑120B, Llama‑4, DeepSeek‑R1, and diffusion models.