EricLBuehler/mistral.rs

Fast, flexible LLM inference

What it solves

mistral.rs is a high-performance LLM inference engine designed to make running large models locally or in production with zero configuration. It removes the friction of manual setup, quantization, and hardware optimization, allowing users to run any Hugging Face model with a single command.

How it works

The engine is built on the Candle framework and supports a wide array of hardware accelerators (CUDA, Metal, CPU). It uses continuous batching, PagedAttention, and FlashAttention V2/V3 to maximize throughput. It provides a zero-config CLI that auto-detects model architectures, quantization formats, and chat templates. For serving, it exposes OpenAI-compatible and Anthropic-compatible APIs, and includes a built-in web UI for interacting with models.

Who it’s for

  • Developers building AI applications using Python or Rust SDKs.
  • MLOps engineers deploying LLMs with Prometheus metrics and Docker support.
  • SREs looking for a high-throughput inference server with multi-GPU and distributed inference capabilities.
  • AI researchers experimenting with multimodal models (text, vision, video, audio) and various quantization methods.

Highlights

  • Zero-Config Execution: Run any Hugging Face model instantly via mistralrs run -m user/model.
  • True Multimodality: Integrated support for text, vision, video, audio, and speech generation in one engine.
  • Agentic Runtime: Built-in support for tool calling, local Python code execution, shell execution, and web search.
  • OpenAI & Anthropic Compatibility: Serves as a drop-in replacement for these APIs, including support for the Anthropic Messages API.
  • Smart Quantization: Automatically selects the best quantization format (e.g., ISQ, GGUF, GPTQ, AWQ) based on hardware.
  • Distributed Inference: Supports multi-GPU and multi-node distributed inference using NCCL TP.
  • Built-in Web UI: Includes a native interface for reasoning, code execution, and file management.