peonist-ai/halogen-flash-server

What it solves

Halogen Flash Server is a high-performance inference engine designed specifically for the Qwen3.8-Flash-Next model family running on AMD Strix Halo hardware. It eliminates general-purpose runtime overhead and portability layers to achieve significantly faster prefill and decode speeds than general-purpose engines, particularly for long-context prompts.

How it works

The project implements custom kernels written specifically for one GPU and one model family, removing all fallback paths and portability layers. It uses speculative decoding (with both the model's own draft head and prompt lookup) to optimize speed without sacrificing output quality. The server provides an OpenAI-compatible API (/v1) and also supports the OpenAI Responses API to enable compatibility with tools like the OpenAI Codex CLI.

Who it’s for

Users with AMD Strix Halo hardware who want the fastest possible local execution of Qwen3.8-Flash-Next, including those requiring high-precision (5.53 bpw) and long-context capabilities (up to 1M tokens).

Highlights

• Hardware-Specific Optimization: Custom kernels tailored for AMD Strix Halo for maximum performance.
• High Speed: Claims roughly 4x faster end-to-end performance than competing runtimes on the same hardware.
• Speculative Decoding: Pure speed optimization that maintains byte-identical output to serial greedy decode at temperature 0.
• OpenAI Compatibility: Supports standard Chat Completions and the Responses API.
• Vision Support: Optional vision tower integration for image processing with configurable resolution and scaling.
• Memory Management: Explicitly manages locked weights and KV pool positions to optimize unified memory usage on 128 GB machines.