gufo-org/gufo

Gufo – a local inference engine for AMD Strix Halo GPUs

What it is – Gufo is a C++‑based inference server that runs large language, vision and audio models on AMD’s Strix Halo accelerator (Ryzen AI MAX+ 395 with Radeon 8060S). It talks the OpenAI chat‑completion API, so existing tools (VS Code, Open WebUI, curl, etc.) can use it just like a cloud service.

Key features

• Hardware‑specific optimisation – kernels are hand‑tuned for the gfx1151 GPU, using ROCm HIP, hipBLAS/hipBLASLt and other AMD libraries. The engine is built for the 128 GiB unified memory configuration but also works on smaller cards.
• Model catalogue – ready‑to‑run GGUF checkpoints for several state‑of‑the‑art models (Qwen 3.8 27B, Qwen 3.8 Flash‑Next, DeepSeek V4 Flash, Qwen 3‑ASR, Qwen 3‑TTS, etc.). Benchmarks and quality reports are provided in the docs/models folder.
• Speculative decoding – supports DFlash2, MTP and DSpark speculative modes that dramatically increase token‑per‑second throughput (e.g., 656 tok/s per‑prompt for Qwen 3.8 27B).
• Multi‑modal support – text, images, audio (ASR/TTS), and experimental video/audio generation.
• Containerised deployment – a pre‑built OCI image (ghcr.io/gufo-org/toolboxes/gufo-runtime) can be run with Podman, exposing the service on port 8080.
• Concurrent request handling – designed for batching, cancellation and conversation caching out of the box.

How to get started

1. Download a model from Hugging Face using the hf download command shown in the README.
2. Pull the runtime container and run it with the required GPU devices (/dev/kfd, /dev/dri).
3. Start the server (gufo serve llm …) pointing at the downloaded GGUF file and, optionally, a speculative‑model file.
4. Query the API with a simple curl POST to http://localhost:8080/v1/chat/completions.

The README also explains how to build Gufo from source (Nix or classic CMake), the required ROCm libraries, and how to handle SELinux or root‑less Podman permissions.

Who it’s for – developers and researchers who own a Strix Halo workstation and want a fast, locally‑hosted AI service without relying on cloud GPUs. It fills the niche of a “local OpenAI‑compatible endpoint” for AMD hardware, similar to what llama.cpp does for CPUs.

License – the core code is MIT‑licensed; third‑party components keep their own notices. Model weights are not bundled and remain under the original publishers’ terms.