SigmanticAI/apex-inference-chip

An inference chip design that runs a real LLM (Qwen2.5-0.5B) on FPGA — one transformer decoder layer in RTL, every silicon value bit-exact against a golden model. 0.56 tok/s measured, a 140× climb, full evidence trail.

What it solves

APEX is designed to address the two primary bottlenecks of LLM inference at the edge: the high cost of moving model weights (bandwidth) and the growing memory overhead of the KV cache (context memory). Unlike traditional accelerators that handle KV cache quantization in software, APEX integrates a compression codec directly into the hardware datapath to maintain consistent reading speeds as context grows.

How it works

The project implements a single transformer decoder layer as a hardware tile. It uses a time-multiplexed INT8 systolic GEMM engine (MXE) to handle all matrix operations in a layer. Key architectural features include:

  • In-flight KV Compression: Keys and values are compressed the moment they are produced and decompressed when consumed, using adaptive precision (INT4 with an fp16 outlier lane) based on token importance.
  • Layer Walker: An on-tile sequencer that fetches descriptors and routes tensors through a routing fabric (XBR), allowing a full layer to execute with a single host command rather than multiple round-trips.
  • Online Softmax: A numerically stable implementation that processes scores as a stream, eliminating the need for large buffers.
  • Weight Streaming: Weights are streamed from DDR via a dedicated fuel line, with 4-bit weights unpacked to INT8 at the feeder to reduce traffic.

Who it’s for

Hardware engineers, FPGA developers, and AI researchers focused on efficient LLM inference, custom silicon design, and hardware-level KV cache optimization.

Highlights

  • Bit-Exact Verification: Every RTL block is verified against an executable NumPy golden model, ensuring zero tolerated mismatches.
  • Hardware Proven: Demonstrated on Lattice ECP5 and AWS F2 (VU47P) FPGA hardware using Qwen2.5 models.
  • Adaptive Precision: Uses a TIP unit to track token importance and drive precision tiers (KVQ8/KVQ4/KVQ4+).
  • Mutation Testing: Employs mutation-tested testbenches to ensure verification suites actually catch RTL bugs.

Related

  • Project
    LMCache/LMCache

    A KV cache management layer for LLM inference that reduces TTFT and improves throughput by making KV caches persistent and reusable across requests and engines.

  • Project
    WeianMao/triattention

    A KV-cache compression method for long-reasoning LLMs that uses trigonometric frequency-domain compression to reduce memory usage by 10.7x and boost throughput by 2.5x without accuracy loss.

  • Project
    jundot/omlx

    An optimized LLM inference server for Apple Silicon Macs featuring tiered KV caching, continuous batching, and a native macOS menu bar management app.

  • Project
    Zefan-Cai/KVCache-Factory

    A unified playground for implementing and evaluating KV cache compression, retrieval, merging, and quantization methods to optimize long-context LLM inference memory usage.

  • Dispatch
    vLLM Performance Benchmarks: Topping Artificial Analysis Leaderboard

    vLLM has achieved top rankings on the Artificial Analysis leaderboard for DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B by implementing aggressive kernel fusion and speculative decoding optimizations.