Running a 28.9M Parameter LLM on an $8 ESP32-S3

Overview

The project demonstrates a 28.9 million‑parameter language model running entirely on an ESP32‑S3 microcontroller that costs about $8.

How the Model Fits in Limited Memory

Most parameters reside in flash as a lookup table, with only a small core in SRAM/PSRAM, enabled by per‑layer embeddings. The ESP32‑S3 provides 512 KB SRAM, 8 MB PSRAM and 16 MB flash. Normally a model must be fully reachable from fast memory, which limits size to a few hundred thousand parameters. By keeping the 25 million‑parameter embedding table in flash and reading only the few rows needed per token (≈450 bytes), the large model costs almost nothing to run. The idea comes from Google's Per‑Layer Embeddings, used in Gemma 3n and Gemma 4, adapted here to a microcontroller memory layout.

Performance Characteristics

The system generates text at about 9.5 tokens per second end‑to‑end, with 9.7 tokens per second of pure compute. The model is stored at 4‑bit precision, occupying 14.9 MB of flash. Parameters: 28.9 M total, 25 M of them in the flash lookup table. Chip: ESP32‑S3 ($8). Connectivity: none, everything runs on the device.

What the Model Can and Cannot Do

Trained on the TinyStories dataset, the model writes short, simple stories and keeps them mostly coherent. It will not answer questions, follow instructions, write code, or know facts. This limitation stems from the small reasoning portion of the model; the memory trick does not change the model’s capabilities.

Running the Demo Yourself

Firmware, wiring, and flashing steps are in firmware/esp32_llm/README.md. Training, ablation, and quantization code live in src/ and experiments/. Full method, ablations, and on‑chip measurements are written up in RESULTS.md.

Community Reaction on Hacker News

Commenters highlighted the achievement, compared it to other low‑cost boards, speculated about TTS use, questioned flash wear and model accuracy, and expressed interest in scaling to larger devices.

"It's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards: https://milkv.io The duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!" — @titzer

"This is a really neat use of the per-layer embedding trick. It's also worth noting that there viable TTS models that are ~20-30M param, so it might mean you can have a ESP32 with no network access read stuff out to you in near real time!" — @rao-v

"Why can't this scale to run much larger models on CPU backed by flash with good access patterns?" — @chrishynes

"Wont this wear out the flash memory quickly? I wonder hiw many read cycles can it survive" — @rvba

"How accurate is this quantized model?" — @caruasdo

"i don't care about microcontroller, what is the decent option to have local llm in my raspi4 that does not take 30 seconds to answer?" — @anonymous344