openTPU: An AI-Developed Open-Source Inference Accelerator

AI-Driven Hardware Design for Inference

openTPU is an open-source AI accelerator designed to demonstrate that AI agents can handle hardware design and build the chips required to run their own inference. The project provides a complete vertical stack in a single monorepo, including the hardware design (SystemVerilog), instruction set architecture (ISA), a bit-exact simulator, a kernel language with its compiler, and the host software required to drive a physical PCIe card.

According to the project author, the development process utilized a recursive self-improvement loop. After initially using AI to develop RISC-V CPU cores, the same technique was applied to openTPU, moving the accelerator from producing only a few tokens per second to over 80 tokens per second on smaller models.

Hardware Specifications and Implementation

openTPU is implemented on an Inspur YPCB-00338 card featuring a Xilinx Kintex-7 xc7k480t FPGA and two DDR3 channels (DDR3-1066 with a 17.1 GB/s peak). The system operates at a clock speed of 133.33 MHz.

Architecture Components

• Sequencer: Issues one instruction per cycle to various functional units.
• DMA: Handles data movement.
• Matrix Unit: A four-column systolic matrix unit that multiplies int8 weights streamed from DRAM.
• Vector Unit: Performs FP32 mathematics.
• Quantizer: Converts results back into int8.
• LiteDRAM Controllers: Calibrated by an internal CPU within the memory core, allowing the card to calibrate both DDR3 channels in 12 seconds without host involvement.

The architecture is designed for simplicity and transparency; there is no cache or hidden scheduling. Every data movement is an explicit instruction, allowing for precise performance profiling via the included "Lens" profiler, which provides roofline and timeline views of hardware utilization.

Performance Benchmarks

openTPU supports a wide range of modern models, including Qwen3, Qwen3.5, LFM2.5, Gemma 4, SmolLM3, and Phi-4-mini. The design achieves bit-for-bit parity between the physical FPGA card and the Python-based ISA simulator.

Decode and Prefill Performance

Performance varies by model size and quantization level (int8 vs. 4-bit).

Model	Weights	Decode (Device)	Prefill (Device)	DRAM Bandwidth
LFM2.5-230M	4-bit, int8 head	85.8 tok/s	335.4 tok/s	14.1 GB/s (82% peak)
Qwen3-0.6B	4-bit, int8 head	31.3 tok/s	103.4 tok/s	13.9 GB/s (82% peak)
Qwen3.5-0.8B	4-bit, int8 head	24.5 tok/s	23.3 tok/s	66.7 tok/s
LFM2-2.6B	4-bit, int8 head	10.96 tok/s	10.93 tok/s	20.6 tok/s
Phi-4-mini (3.8B)	4-bit, int8 head	6.56 tok/s	15.0 tok/s	15.8 GB/s (92% peak)
Qwen3.5-4B	4-bit, int8 head	5.88 tok/s	12.9 tok/s	15.7 GB/s (92% peak)

Memory Management and Quantization

• 4-bit Quantization: Uses FP4 values with two-level block scales (4.25 bits per weight) while keeping the LM head in int8. This reduces bytes per token by approximately one-third and increases decode speed by 40% to 45%.
• Expert Offloading: For Mixture-of-Experts (MoE) models exceeding the card's 4 GiB capacity, experts are streamed from host storage. For example, Qwen3.5-35B-A3B achieves 3.95 tok/s with 62% of expert uses hitting the local DRAM slots, streaming 153 MB per token over PCIe at 1.41 GB/s.

Software Stack and Tooling

The project uses a custom kernel language (ol) with a JIT compiler that translates high-level operations (like ol.dot and ol.quantize) into the 32-bit word ISA.

Tooling Suite

• otpu-chat: Interface for chatting with supported models using either the ISA simulator or the physical board.
• otpu-smi: System Management Interface for monitoring temperature, power, DRAM bandwidth, and unit utilization.
• otpu-lens: A browser-based profiler that records runs from the RTL, simulator, or card to analyze cycle-by-cycle execution.

Community Perspectives

While the project demonstrates a functional AI-designed accelerator, community discussion on Hacker News highlights a debate regarding the definition of "AI-developed hardware."

"AI is now capable of developing its own inference hardware No, it isn't. A human prompted an LLM to build a software simulation environment for hardware design, enabling an LLM, when prompted by a human, to optimize hardware designs against constraints in the simulation."

Other contributors noted the educational value of the open-source nature of the project, which allows users to trace a matrix multiplication from Python code down to the physical wires of the FPGA.