luminal-ai/luminal

Inference at the speed of light.

What it solves

Luminal is a high-performance inference compiler designed to eliminate the complexity and performance overhead found in traditional deep learning frameworks. It addresses the "complexity explosion" of traditional compiler stacks (like XLA or TVM) and the inefficiency of eager-first execution (like PyTorch) by treating all neural networks as static computation graphs that are compiled ahead-of-time for maximum hardware efficiency.

How it works

Luminal uses a RISC-style architecture, reducing all operations to 15 primitive ops. Instead of relying on hand-written heuristics or destructive rewrite rules, it employs a search-based approach to discover the most efficient execution path and optimizations (such as Flash Attention) automatically. It is written in Rust and interacts directly with accelerator APIs (CUDA, Metal) to avoid abstraction layers. It also supports symbolic dimensions to handle dynamic shapes while maintaining the ability to perform aggressive specialization.

Who it’s for

It is intended for developers and researchers who need maximum inference performance on any device, specifically those looking for a lightweight, statically-linked Rust alternative to heavy ML frameworks or those wanting to use PyTorch models with a high-performance compiler backend.

Highlights

  • PyTorch Integration: Works as a torch.compile backend, allowing users to compile PyTorch models directly.
  • Extreme Performance: Capable of running Q8 Llama 3 8B at approximately 80% of the theoretical maximum performance on an H100.
  • Search-Based Optimization: Automatically discovers complex optimizations without needing manual heuristics.
  • Minimalist Core: Designed to be simple enough that the entire core library can be understood in a single afternoon.
  • Native Rust: A statically-linked crate that avoids Docker containers and virtual environments.

Related

  • Project
    huggingface/candle

    A minimalist ML framework for Rust that enables lightweight, Python-free serverless inference and browser-based model execution.

  • Project
    EricLBuehler/mistral.rs

    A fast, flexible LLM inference engine written in Rust that supports multimodal models, agentic tool execution, and OpenAI/Anthropic compatible APIs.

  • Project
    lablup/mlxcel

    A high-performance native runtime and server for LLMs, VLMs, and audio models on Apple Silicon and NVIDIA CUDA, removing Python from the inference path.

  • Project
    guoqingbao/xinfer

    A blazing-fast LLM inference engine written in pure Rust that eliminates Python/PyTorch dependencies to enable high-throughput, production-ready model serving on consumer and professional GPUs.

  • Project
    pegainfer-project/pegainfer

    PegaInfer is a Rust‑based, CUDA‑accelerated LLM inference server that provides an OpenAI‑compatible API, supports multiple model families (Qwen3, Gemma 4, DeepSeek‑V2‑Lite, etc.), and offers production‑grade performance without any Python runtime.