peonist-ai/halogen-flash-server

Local inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models, over the OpenAI and Anthropic APIs on one port.

halogen‑flash‑server – ultra‑fast inference for Qwen 3.8‑Flash‑Next on AMD Strix Halo

What it is – A purpose‑built server that runs the Qwen 3.8‑Flash‑Next large language model on a single AMD Strix Halo GPU. The code contains custom kernels that are tuned for that exact silicon and model family, so it can start answering a request in ~1.6 s and sustain very high token‑throughput (up to 94 tokens / s for repetitive text, >1 800 tokens / s for long pre‑fills).

Why it matters – General‑purpose runtimes (e.g., vLLM, llama.cpp) must support many GPUs and models, which adds overhead. By stripping away any portability layer and writing the kernels for one GPU only, halogen achieves speeds that other servers cannot match, making it attractive for AI agents that need low latency over very long context windows (30 k–96 k tokens).

Key capabilities (as described in the README)

Feature	Detail
Model support	Qwen 3.8‑Flash‑Next (v2 checkpoint) out‑of‑the‑box; also can load a compatible `llama.cpp` GGUF file (e.g., Unsloth’s `UD‑IQ4_XS`).
Throughput	Prefill: 1 821 tok/s (8 k prompt) → 1 675 tok/s (131 k prompt). Decode: up to 94 tok/s for repetitive output; 73 tok/s for code, 54 tok/s for prose.
Latency	First token of an agent turn arrives in ~1.6 s; full turn (≈200 token answer) finishes in ~4.2 s. Cold start for a 34 k‑token prompt: ~21 s.
Context length	Supports up to 1 M tokens (opt‑in) with special configuration; typical runs use 32 k–131 k token prompts.
Precision	4.16 bits per weight (bpw) for the default checkpoint; 5.25 bpw when counting all 179.55 B parameters with FP8 n‑gram lookup.
Speculative decoding	Uses draft heads (model‑provided or request‑provided) to speed up greedy decode without sacrificing output quality.
API	OpenAI‑compatible HTTP endpoints (completion, chat, embeddings, classifier, image generation, etc.). Detailed per‑endpoint docs in `docs/api.md`.
Configuration	Cache modes, KV‑pool sizing, disk‑based prompt cache, token‑budget limits, optional API‑key enforcement, request logging.
Deployment	Distributed as a container image (`ghcr.io/peonist-ai/halogen-flash-server`). Quickstart script shows how to run with Podman (or Docker) exposing the GPU via `/dev/kfd` and `/dev/dri`.
Hardware requirement	AMD Ryzen AI Max+ 395 (Strix Halo) with at least 128 GB RAM; BIOS UMA frame‑buffer must be set to minimum so the engine can use host RAM instead of dedicated VRAM.

Typical use‑case workflow

Prepare the host – Set BIOS UMA carve‑out to minimum, install container runtime, ensure /dev/kfd and /dev/dri are accessible.
Pull the image – podman pull ghcr.io/peonist-ai/halogen-flash-server:0.18.1.
Run the container – Use the Quickstart command (mount a local ~/halogen-models directory for the model weights, expose port 8731, pass HALOGEN_DOWNLOAD env var to fetch the official Qwen weights).
Call the API – Send OpenAI‑style requests (e.g., POST /v1/chat/completions) from any client library. For agents, send the full conversation history each turn; the server will return the answer in a few seconds.
Tune – Adjust cache mode, KV‑pool size, or enable the 1 M‑token context via docs/configuration.md if you need longer windows.

Getting started resources

Docs site – https://peonist-ai.github.io/halogen-flash-server/ (searchable).
Quickstart – docs/getting-started.md (the README already includes the one‑liner Docker/Podman command).
Agent guide – AGENTS.md explains how to hook the server into an autonomous coding agent.
Performance numbers – docs/performance.md reproduces the throughput/latency figures with a single command.
Troubleshooting – docs/troubleshooting.md covers common start‑up failures, SELinux issues, and long‑prompt hangs.

Community & support

Discord – https://discord.gg/bcm6QknaV6 – real‑time help, benchmark sharing, hardware‑specific tips.
GitHub Issues – for bugs and regressions; the repo includes a CONTRIBUTING.md that tells you what information to provide.
Security – reported directly to the maintainers (no public issue).

License

The engine itself is released under the license in LICENSE.md (a permissive open‑source license). Third‑party components are listed in THIRD-PARTY-NOTICES.md. Model weights are not bundled; they remain under the original authors’ license and must be obtained separately (the README’s Quickstart pulls them from the peonist-ai/halogen-qwen3.8-flash-next image).

Bottom line

halogen‑flash‑server is a highly specialized inference server that squeezes maximum speed out of a single AMD Strix Halo GPU for the Qwen 3.8‑Flash‑Next model. It is aimed at developers building AI agents that need sub‑second first‑token latency and can afford a hardware‑locked deployment. The project provides thorough documentation, a ready‑to‑run container image, and an active Discord community for support.