warpfront/hipfire

RDNA-native LLM inference engine in Rust.

hipfire – LLM inference on AMD GPUs

What it is – hipfire is a native inference engine that runs large language models (LLMs) directly on AMD GPUs. It is written in Rust, uses the HIP API and a custom dispatch layer called Redline. No Python, PyTorch or CUDA‑translation layers are involved in the hot path, so the whole stack stays in native AMD code.

Why it matters – AMD GPUs have historically lagged behind NVIDIA for LLM work because most open‑source runtimes target CUDA. hipfire fills that gap by providing a production‑ready, OpenAI‑compatible server that can serve quantised models on RDNA and CDNA hardware, with support for multi‑GPU expert parallelism, speculative decoding, and even the XDNA2 NPU on newer “Strix Halo” GPUs.

Key capabilities

Feature Details
Rust‑only runtime The inference loop, model loader and CLI are all Rust; the only external dependency at runtime is ROCm, loaded dynamically.
Quantisation formats Supports several custom formats (MQ3‑MQ6, MQ4R, MFP4) that trade off size vs. speed.
Model registry Bundles a JSON registry with 90 curated model entries (Qwen, DeepSeek, Flux, etc.) and 82 aliases.
CLI & server hipfire pull, run, chat, serve, bench, quantize plus an OpenAI‑compatible HTTP daemon (default port 11435).
Speculative decoding DFlash draft models and native MTP heads for faster token generation.
Multi‑GPU expert parallelism EP2 (two‑GPU) and EP4 routes for MoE models; automatic owner‑routed pre‑fill.
NPU support On gfx1151 (Strix Halo) can off‑load the trunk projection to the XDNA2 NPU.
PeaceMaker toolchain In‑tree kernel builder that emits and certifies AMD‑GPU and NPU machine code without invoking hipcc.
Redline replay layer Records the kernel graph, validates it, and replays it for deterministic performance; falls back to normal HIP dispatch if validation fails.
OpenAI‑compatible API Serves /v1/chat/completions etc., with optional flags like enable_thinking.
Extensive docs Over 20 markdown pages covering install, model selection, quantisation, configuration, benchmarking, GPU matrix, and more.

Getting started (quick example)

# Grab a 27‑billion‑parameter Qwen model (MQ4‑XTS quantisation)
hipfire pull qwen3.8:27b-mq4-xts

# Start a local server (listens on 127.0.0.1:11435)
hipfire serve qwen3.8:27b-mq4-xts -d

# Chat with it from the CLI
hipfire chat qwen3.8:27b-mq4-xts

For a single‑shot request you can also run:

hipfire run qwen3.8:27b-mq4-xts "What is the capital of France?"

The same model can be accessed via curl using the OpenAI‑compatible endpoint.

Model support

The built‑in registry includes many families:

  • Qwen 3.8 / 3.6 / 3.5 (dense, MoE, Flash drafts) – sizes from 0.8 B up to 35 B.
  • Ornith 1.5, Muse Glimmer, DeepSeek V4 Flash, MiniMax‑M2.7, North‑Mini‑Code, Carnice, Qwopus, LFM2.5, VibeThinker, etc.
  • Flux image generation models (flux.schnell:1). All weights are hosted on the hipfire-models HuggingFace organization. Users can also import their own models via hipfire quantize.

GPU coverage

Architecture family GPUs (examples) Highlights
Vega / CDNA gfx906, gfx908, gfx940‑942 Wave64 HIP kernels, DP4A‑accelerated pre‑fill for older cards.
RDNA1 gfx1010‑1013 Portable HIP + Redline dispatch.
RDNA2 gfx1030‑1032 Same as RDNA1 with newer ISA.
RDNA3 gfx1100‑1103 WMMA kernels, MQ4R route.
RDNA3.5 gfx1150‑1152 WMMA + optional XDNA2 NPU pre‑fill (Strix Halo).
RDNA4 gfx1200, gfx1201 WMMA, retained‑PM4 Redline, EP2 multi‑GPU parallelism.

The engine selects the optimal kernel per‑architecture; if a specialized path is missing it falls back to a portable implementation.

Installation

  • Linux (ROCm 6+). One‑line installer pulls a pre‑built kernel pack or builds locally with hipcc:
curl -fsSL https://raw.githubusercontent.com/warpfront/hipfire/v0.4.2/scripts/install.sh | bash
  • Windows – source build instructions are in GETTING_STARTED.md.
  • NixOS – provides a flake and module for declarative installation.
  • Containers – a Containerfile builds a runtime image; run with /dev/kfd and /dev/dri devices exposed.

Uninstall with the matching script; --purge also removes ~/.hipfire.

Documentation & community

  • Docs – a full table of contents is in the README; topics include model selection, quantisation, configuration, serving, multi‑GPU parallelism, image generation, architecture details, benchmarking methodology, and validation rules.
  • Discord – active chat at https://discord.gg/F3BaywB8Rs.
  • PeaceMaker – the kernel toolchain is also released as a separate repo (warpfront/peacemaker).

License

hipfire is Apache‑2.0 (v0.3.0 onward); earlier releases were dual‑licensed MIT/Apache‑2.0. Some source files retain MIT with the LLVM exception.

TL;DR

hipfire is a production‑grade, Rust‑based LLM inference engine that runs directly on AMD GPUs (RDNA 1‑4, CDNA) without any Python or CUDA layers. It ships a curated model registry, a feature‑rich CLI, an OpenAI‑compatible server, and a custom kernel toolchain (PeaceMaker) plus a replay‑optimised dispatcher (Redline). If you have an AMD GPU and want to run large language models locally, hipfire is the most complete open‑source solution available today.

Related

  • Project
    kkokosa/dotLLM

    A high-performance LLM inference engine written natively in C#/.NET, providing native CPU/GPU support for transformer models without relying on Python or llama.cpp.

  • Project
    EricLBuehler/mistral.rs

    mistral.rs is a Rust‑based engine for fast, flexible local LLM inference. It supports many model formats (Hugging Face, GGUF), automatic hardware‑aware quantization, multimodal inputs, and built‑in agentic features (tool calling, Python/shell execution, web search). It can serve OpenAI‑compatible and Anthropic‑compatible APIs, offers a web UI, and provides Rust and Python SDKs for embedding.

  • Project
    guoqingbao/xinfer

    A blazing-fast LLM inference engine written in pure Rust that eliminates Python/PyTorch dependencies to enable high-throughput, production-ready model serving on consumer and professional GPUs.

  • Project
    lablup/mlxcel

    A high-performance native Rust runtime for LLM, VLM, and audio inference on Apple Silicon, CUDA, and ROCm, removing Python from the production request path.

  • Project
    Avarok-Cybersecurity/atlas

    A pure Rust and CUDA LLM inference engine that provides a lightweight, high-throughput server for large models without requiring Python or PyTorch.