TinyStories LLM on a $250 KV260 FPGA achieves 60k tokens/s

Takeaway

A 3.16 M‑parameter INT4 LLM fits in the 3 MB on‑chip SRAM of a $250 AMD KV260 FPGA and delivers a measured 59,965 tokens per second (200 MHz, 16 parallel streams) and a usable 19,200 tokens per second (single‑stream, full KV cache) in a live web demo, demonstrating that on‑chip weight storage can overcome the DDR bandwidth wall for tiny models.

Why on‑chip memory matters

The KV260’s ARM A53 cores and programmable logic (PL) share a single off‑chip DDR controller limited to ~20 GB/s. When model weights reside in DDR, every token generation incurs a costly round‑trip over the AXI bus, making the system memory‑bound rather than compute‑bound. Storing the entire weight matrix in on‑chip SRAM (URAM ~18 Mb + BRAM ~5 Mb) raises effective bandwidth to hundreds of GB/s, eliminating the bottleneck. This insight mirrors the approaches of Taalas, Groq, and Cerebras, which all allocate large on‑chip memory budgets to escape the memory wall.

Model fitting strategy

Model size – 3.16 M parameters × 4 bits = ~1.5 MiB, comfortably within the KV260’s ~3 MiB on‑chip budget.
Weight format – INT4 quantisation provides the required density while keeping arithmetic simple.
Corpus – TinyStories (≈2.1 M simple children’s stories) supplies sufficient linguistic variety for a toy chat model.
Lemmatised variant – A Kevin‑style preprocessing step reduced the corpus length by ~30 % (371.7 M → 260.5 M words) but did not shrink the parameter count; it only speeds up generation by producing fewer characters per token.

Architecture and implementation

Dual‑ported URAM – Holds the weight image and enables a “split‑brain” design where two cohorts read weights simultaneously.
BRAM – Stores activations, scratch space, and the KV cache.
Custom GEMV – Wide‑word matrix‑vector multiplication implemented directly in Verilog, avoiding any high‑level synthesis (HLS) tools.
Parallel streams – Up to 16 independent token streams share a single weight pass, each using an attention window of T = 1 to minimise round‑trips. This yields the headline 59,965 tok/s aggregate rate.
Full‑context stream – A single stream with the complete KV cache (attention over all prior tokens) provides a usable chat at ~19,200 tok/s (counted cycles) and ~21,300 tok/s measured live.

Performance ladder

Stage	Configuration	Tokens / s (silicon)	Notes
A53 baseline	ARM A53 only	11	CPU‑only, DDR‑bound
GPU reference	RTX 3050 Ti (torch)	719
Single‑stream PL	Full forward in fabric, full KV cache	19,242 (counted) ≈ 21,300 (live)
4‑16 parallel streams	T = 1, no KV cache	59,965 (16‑stream aggregate)

The ladder shows that each optimisation removes a specific source of latency: moving matmul to PL, eliminating CPU‑fabric handshakes, and finally aggregating weight passes across parallel streams.

Limits and scalability

On‑chip capacity – The KV260 provides ~3 MiB; models larger than ~6.3 M parameters would spill into DDR, re‑introducing the bandwidth wall.
DSP resources – The chip has 1,248 DSP48E2 blocks, each capable of two INT4×INT8 MACs. Packing the required MACs for a larger model would exceed available DSPs, capping practical throughput at 62k–78k tok/s for this architecture.
Usability – The 16‑stream aggregate speed is achieved with an attention window of one token, producing nonsensical output. The single‑stream, full‑context mode is the only configuration that yields coherent dialogue, albeit at ~21k tok/s.

Comparison to ASIC and larger‑scale solutions

Taalas ASIC – Stores weights in ROM, achieving higher absolute throughput but lacking post‑fabric reprogramming.
Cerebras Wafer‑Scale Engine – Provides 44 GB SRAM per wafer, enabling much larger models without DDR bottlenecks.
Groq – Allocates ~230 MB on‑chip SRAM per chip, supporting multi‑hundred‑megabyte models.

The KV260 demonstrates that the same memory‑wall principle applies across budgets: on‑chip weight storage yields orders‑of‑magnitude speedups even on a $250 development board.

Community feedback highlights

"The problem is not that your model is fast. GPUs scale well for both training and inference, and the technical floor is low. FPGA design requires deep hardware expertise." – stevefan1999

"I didn't expect the 2,000‑connection sweep to stay flat, since all of them are sharing one stream. What does per‑user latency look like?" – haeseong

"Conceptually it's a cool idea. Practically the results seem about as coherent as import random; print(random.choice(list(my_dict)))… is there a practical use to a model this small?" – serf

"Really cool project. I wonder what the future of LLM inference will look like… emerging memory technologies may be needed for larger models." – variadix

These comments underline the trade‑off between raw speed and model quality, the steep learning curve of FPGA development, and the broader question of how on‑chip memory innovations will shape future LLM inference.

How to reproduce

The full source—including the Verilog RTL, the Keviniser preprocessing script, the INT4‑quantised TinyStories model, and the serving stack—is available at github.com/MichaelAyles/kev-gpt. The build flow uses Xilinx Vivado CLI on a 20‑core i7; a complete synthesis, place‑and‑route, and bitstream generation takes ~30 minutes.

Bottom line

Storing a tiny INT4 LLM entirely in on‑chip SRAM of a low‑cost FPGA removes the DDR bandwidth wall and yields a measured 60 k tokens / s aggregate throughput, while a single‑stream, full‑context configuration still delivers a usable ~21 k tokens / s under heavy load. The experiment validates the core hypothesis that on‑chip weight storage—regardless of budget—can dramatically accelerate inference for models that fit within the memory budget.