huggingface/candle

Minimalist ML framework for Rust

What it solves

Candle is a minimalist machine learning framework for Rust designed to enable serverless inference and remove Python from production workloads. It addresses the overhead and large binary sizes associated with full frameworks like PyTorch, allowing for the deployment of lightweight, fast binaries.

How it works

It provides a PyTorch-like syntax for tensor operations and model training. Candle supports multiple backends, including an optimized CPU backend (with MKL and Accelerate support), a CUDA backend for GPU acceleration, and WASM for running models directly in the browser. It can load models from various formats including safetensors, npz, ggml, and PyTorch files.

Who it’s for

Developers who want to build and deploy high-performance ML models in Rust, specifically those looking to avoid Python's GIL and the heavy footprint of traditional ML frameworks for serverless or browser-based deployments.

Highlights

  • Broad Model Support: Includes implementations for LLaMA, Mistral, Mixtral, Stable Diffusion, Whisper, and many others.
  • Multi-Backend: Native support for CPU, CUDA, and WebAssembly (WASM).
  • Lightweight: Focused on creating small binaries for efficient serverless deployment.
  • PyTorch-like API: Familiar syntax for those coming from PyTorch, making the transition to Rust easier.
  • Quantization: Supports quantization using llama.cpp types to reduce model size and memory usage.

Related

  • Project
    luminal-ai/luminal

    A high-performance general-purpose inference compiler written in Rust that optimizes ML models by treating them as static computation graphs and using search-based optimization.

  • Project
    pegainfer-project/pegainfer

    PegaInfer is a Rust‑based, CUDA‑accelerated LLM inference server that provides an OpenAI‑compatible API, supports multiple model families (Qwen3, Gemma 4, DeepSeek‑V2‑Lite, etc.), and offers production‑grade performance without any Python runtime.

  • Project
    SomeB1oody/RustyML

    A pure Rust machine learning and deep learning library that provides a complete workflow from data preprocessing to model evaluation without C/C++ dependencies.

  • Project
    mosecorg/mosec

    A high-performance model serving framework that uses a Rust core and Python interface to turn ML models into efficient, cloud-ready microservices.

  • Project
    lucasjinreal/Crane

    Crane is a Rust‑based inference framework (built on candle) that runs LLMs, vision‑language models, OCR, ASR, TTS and music transcription fast on CPU, CUDA, Metal and experimental ROCm. It loads GGUF checkpoints directly, provides a clean Rust API and an OpenAI‑compatible HTTP server, and ships ready‑to‑use demo binaries for chat, image chat, OCR, speech, etc.