openTPU – An Open‑Source AI Accelerator

What it is

openTPU is a full‑stack, open‑source AI accelerator built around a modest Xilinx Kintex‑7 FPGA card. The repository contains:

• Hardware RTL (SystemVerilog) for the accelerator core.
• Instruction‑set architecture (ISA) and a bit‑exact Python simulator.
• Kernel language & compiler that turn high‑level Python kernels into ISA instructions.
• Host software (otpu‑chat, otpu‑serve, otpu‑smi, otpu‑lens, etc.) that drives the FPGA over PCIe and provides a usable inference interface.
• Documentation, tests and scripts for validation, performance measurement and profiling.

The project is both a real accelerator (it runs on a physical PCIe card) and a learning platform – the entire stack is small enough to be read from top‑to‑bottom.

---

Why it matters

• End‑to‑end openness – unlike most commercial AI chips, every layer—from the silicon RTL up to the Python‑level inference API—is open source.
• Hardware‑aware AI research – the design lets researchers experiment with instruction scheduling, memory layout, quantisation, and even AI‑generated hardware design (the repo mentions the auto‑arch‑tournament project that explores AI agents designing hardware).
• Real‑world performance – on a Kintex‑7 board the accelerator runs a range of modern LLMs (from 230 M to 35 B parameters) at token‑rates comparable to low‑power GPUs, with DRAM utilisation close to the DDR3 peak.
• Educational value – the monorepo is deliberately simple: a single‑issue‑per‑cycle sequencer, explicit DMA instructions, and a visual profiler (Lens) that shows exactly where cycles are spent.

---

How it works (high‑level)

Python kernel → otpu.language (ol) → JIT → ISA instructions →
RTL (matrix unit, vector unit, DMA, quantiser) → FPGA card → PCIe ↔ Host

1. Kernels are written in a tiny Python DSL (opentpu.language).
2. The compiler (@ol.jit) lowers them to the 8‑word, 32‑bit ISA.
3. The ISA simulator (opentpu.isasim.py) provides a reference model.
4. On hardware, the RTL implements a simple sequencer that executes one instruction per cycle, moving data via DMA, performing int8 matrix multiplies, fp32 vector math, and quantising results back to int8.
5. The host tools load a single bitstream that supports all models, manage DRAM, and expose:
   • otpu-chat – an interactive CLI chat.
   • otpu-serve – an OpenAI‑compatible API server.
   • otpu-smi – runtime stats (temperature, power, DRAM bandwidth).
   • otpu‑lens – a profiler that visualises execution on the floor‑plan.

---

Performance highlights (from the README)

Model	Weight format	Decode (device)	Decode (wall)	DRAM bw
LFM2.5‑230M	int8	59 tok/s	52 tok/s	14.5 GB/s (85 % peak)
LFM2.5‑230M	4‑bit	85.8 tok/s	82.1 tok/s	14.1 GB/s
Qwen3‑0.6B	int8	21.6 tok/s	21.3 tok/s	14.4 GB/s
Gemma 4 E2B	4‑bit head	12.14 tok/s	12.09 tok/s	15.5 GB/s
Phi‑4‑mini (3.8B)	int8	3.99 tok/s	3.98 tok/s	16 GB/s
Qwen3.5‑35B (MoE)	4‑bit + int8 head	3.95 tok/s (experts streamed)	–	1.41 GB/s PCIe

Key points: 4‑bit quantisation (FP4 + block scales) cuts weight traffic by ~⅓ and speeds decode by 40‑45 % with modest accuracy loss. DRAM utilisation stays at 80‑95 % of the DDR3 peak, showing the design is memory‑bound rather than compute‑bound.

---

Getting started (quick steps)

# Install the Python stack (simulator works on any laptop)
pip install -e .
pip install pytest torch transformers
python -m pytest -q   # run RTL‑aware tests (needs Verilator)

# Run a model on the simulator
hf download LiquidAI/LFM2.5-230M --local-dir models/LFM2.5-230M
otpu-chat --model lfm2 --backend isa   # interactive chat in the simulator

For the physical card:

make bit               # build the FPGA bitstream (boards/ypcb-00338)
sudo otpu-setup       # program the card via JTAG
otpu-chat --backend board   # now talks to the real accelerator

The README lists other host commands (otpu-serve, otpu-smi, otpu-lens, otpu-selftest).

---

Who might use it?

• Researchers exploring new quantisation schemes, instruction sets, or memory‑layout optimisations for LLM inference.
• Educators & students who want a concrete, open hardware example of an AI accelerator.
• Hobbyists interested in FPGA‑based inference and in running LLMs on inexpensive PCIe cards.
• AI‑hardware developers looking for a reference implementation of a simple systolic matrix engine with explicit DMA scheduling.

---

Limitations / open questions

• The hardware is limited to a Kintex‑7 (≈ 200 mm², DDR3) – performance will not match modern ASICs.
• Models larger than the 4 GiB on‑card DRAM need expert‑offloading; this works but adds PCIe traffic.
• Accuracy for 4‑bit weights is lower than int8; the README provides detailed KL‑loss numbers.
• The project assumes familiarity with FPGA toolchains (Vivado) for building the bitstream.

---

TL;DR

openTPU is a fully open, FPGA‑based LLM inference accelerator that ships a complete hardware description, ISA, simulator, compiler, and host tools. It runs dozens of modern models on a modest Kintex‑7 board, achieving tens of tokens per second while exposing the entire stack for learning and research.