Kaden-Schutt/hipfire

RDNA-native LLM inference engine in Rust.

What it solves

hipfire provides high-performance LLM inference specifically optimized for AMD RDNA GPUs (consumer, pro, and APUs). It addresses the difficulty of running models on consumer AMD hardware, where standard tools like llama.cpp with ROCm can be cumbersome and often treat consumer cards as second-class citizens compared to datacenter hardware.

How it works

Built with Rust and HIP, the project is distributed as a single binary that eliminates Python and PyTorch from the runtime hot path. It uses pre-compiled kernel blobs and JIT-compilation via HIP to target the entire RDNA family (RDNA1 through RDNA4). The engine features a unified dispatch system for kernels (GEMM, attention, MoE, etc.) and supports advanced techniques like speculative decoding (DFlash) and expert-parallel serving for multi-GPU setups.

Who it’s for

Users with AMD RDNA GPUs who want a streamlined, "Ollama-style" experience for running LLMs locally with maximum hardware utilization and minimal software overhead.

Highlights

  • Broad Hardware Support: Targets RDNA1, RDNA2, RDNA3, and RDNA4 across consumer, pro, and APU variants.
  • High Performance: Significant decode speedups over Ollama on RDNA3 hardware (e.g., 7900 XTX).
  • Speculative Decoding: Implements DFlash to drastically increase token generation speed for specific models.
  • Multi-GPU Support: Expert-parallel serving allows sharding large MoE models (like DeepSeek V4 Flash) across multiple GPUs.
  • Memory Management: Includes CASK-based KV cache eviction to prevent Out-of-Memory (OOM) errors during long-context prompts.
  • Developer Friendly: Provides an OpenAI-compatible API and a simple CLI for pulling, running, and serving models.