hyperionaidev/smolmaxxing
maxxing the ability to run different LLMs on the smallest devices possible
smolmaxxing — Run Big AI Models on Tiny Hardware
What it is
smolmaxxing is a project that makes large language models (LLMs) run on very weak devices — TV boxes, old tablets, single-board computers, and routers. The goal: "Run the biggest language model you can on the smallest hardware you own." It covers models from 260K to 16B parameters.
The problem it solves
Most small devices (TV boxes, old tablets, routers) have a CPU, some RAM, and flash storage that sit idle most of the day. Normally, running a modern AI model requires a powerful computer with lots of memory. smolmaxxing figures out the largest model each weak device can actually run, then pushes further with custom code, careful memory use, and streaming weights from flash storage. Every performance number in the repo was measured on the actual device.
Devices tested so far
| Xiaomi TV Box S (3rd Gen) | iPad Air (2013) | |
|---|---|---|
| Chip | Amlogic S7D, 4× Cortex-A55 | Apple A7, 2 cores |
| RAM | 1.9 GB (~1 GB usable) | 1 GB (~700 MB usable) |
| OS | Android 14, no root | iOS 12.5.7, jailbroken |
| Everyday model | Qwen3.5-2B at 7.7 tokens/sec | Bonsai-1.7B (1-bit) at 24–26 tokens/sec |
| Biggest model | Ling-mini-2.0, 16B, at 5.1 tokens/sec | Ling-mini-2.0, 16B, at 2.6–4.5 tokens/sec |
Both devices run 8B and 16B mixture-of-experts (MoE) models on ~1 GB of RAM by keeping only part of each model in memory and reading the rest from flash as needed.
How it works
Everything is a patch on llama.cpp (a popular open-source AI inference engine). Four main techniques:
Kernels built for the chip — Custom low-level code optimized for each processor. A 1-bit lookup-table kernel, repacked 2-bit and ternary formats for the Cortex-A55, and versions for the Apple A7 (which lacks dot-product instructions).
Less RAM per model — Most of a small model's vocabulary is rarely used, so those output rows are dropped. On the TV box, a "headless" mode also stops the launcher and Google services while a model runs.
Experts paged from flash — MoE models only use a few "experts" per token. The project keeps popular experts in RAM, reads others from flash on demand, and prefetches the next layer's experts. This is how 8B and 16B models run on 1 GB of RAM.
Streaming for dense models — For models bigger than RAM, weights stream from flash at the flash's full speed.
How to try it
The quickest start is a ready-made model file from Hugging Face. Important: these files need the patched llama.cpp — stock llama.cpp can't load them.
- TV box or other ARMv8.2 Android device: see
docs/tv-box-setup.md - Jailbroken iPad on iOS 12: see
ipad-air/README.md - All model files and how to rebuild them: see
docs/models.md
What's in the repo
| Path | Contents |
|---|---|
patches/ |
Every change to llama.cpp for the TV box (iPad's patch is in ipad-air/patches/) |
tools/ |
Model rebuilding, GGUF rewriting, benchmarks, kernel checks |
tv-llm-ui/ |
Live token-stream chat page and its launcher |
ipad-air/ |
iOS 12 build, smolchat, deploy and benchmark scripts |
bench/ |
Every chat benchmark (results.jsonl) and raw logs |
data/ |
Vocabulary lists and evaluation texts |
tiny-llm-web/ |
First experiment: a 260K-parameter story model |
zigcc/ |
Zig wrappers for cross-compiling static aarch64 binaries |
Documentation
docs/tv-box.md— every model run on the TV box, speed and qualityipad-air/README.md— iPad Air results, setup, scriptsdocs/how-it-works.md— kernels, memory tricks, expert pager, streaming enginedocs/models.md— downloads, rebuild targets, checksumsdocs/lessons.md— what worked and what didn'tEXPERIMENT.md— full lab notebook: 160 experiments, failures included
License and credits
- License: MIT. The llama.cpp patches are derivatives of llama.cpp (MIT). Model weights keep their own licenses; evaluation texts are Wikipedia excerpts (CC BY-SA 4.0).
- Built on: llama.cpp and llama2.c.
- Models from: Qwen, unsloth, PrismML (Bonsai), inclusionAI (Ling-mini-2.0), tablinum (Moon-Beam), bartowski, mradermacher, Google (Gemma), and karpathy's stories260K.
- Other: The iPad is jailbroken with Amethyst; banner uses Archivo font (SIL OFL); 🤏 logo is from Google's Noto Emoji (Apache 2.0).
Key takeaway
smolmaxxing proves that with clever engineering — custom kernels, aggressive memory trimming, and streaming model weights from flash — you can run surprisingly large AI models (even 16B-parameter ones) on devices with only ~1 GB of usable RAM. The techniques are portable: the 1-bit kernel needs only basic NEON instructions, and flash streaming works on any Linux or Android device with more model than RAM.
Related
- Project
- Project
- Project
- Project