hyperionaidev/smolmaxxing

maxxing the ability to run different LLMs on the smallest devices possible

smolmaxxing — Run Big AI Models on Tiny Hardware

What it is

smolmaxxing is a project that makes large language models (LLMs) run on very weak devices — TV boxes, old tablets, single-board computers, and routers. The goal: "Run the biggest language model you can on the smallest hardware you own." It covers models from 260K to 16B parameters.

The problem it solves

Most small devices (TV boxes, old tablets, routers) have a CPU, some RAM, and flash storage that sit idle most of the day. Normally, running a modern AI model requires a powerful computer with lots of memory. smolmaxxing figures out the largest model each weak device can actually run, then pushes further with custom code, careful memory use, and streaming weights from flash storage. Every performance number in the repo was measured on the actual device.

Devices tested so far

Xiaomi TV Box S (3rd Gen) iPad Air (2013)
Chip Amlogic S7D, 4× Cortex-A55 Apple A7, 2 cores
RAM 1.9 GB (~1 GB usable) 1 GB (~700 MB usable)
OS Android 14, no root iOS 12.5.7, jailbroken
Everyday model Qwen3.5-2B at 7.7 tokens/sec Bonsai-1.7B (1-bit) at 24–26 tokens/sec
Biggest model Ling-mini-2.0, 16B, at 5.1 tokens/sec Ling-mini-2.0, 16B, at 2.6–4.5 tokens/sec

Both devices run 8B and 16B mixture-of-experts (MoE) models on ~1 GB of RAM by keeping only part of each model in memory and reading the rest from flash as needed.

How it works

Everything is a patch on llama.cpp (a popular open-source AI inference engine). Four main techniques:

  1. Kernels built for the chip — Custom low-level code optimized for each processor. A 1-bit lookup-table kernel, repacked 2-bit and ternary formats for the Cortex-A55, and versions for the Apple A7 (which lacks dot-product instructions).

  2. Less RAM per model — Most of a small model's vocabulary is rarely used, so those output rows are dropped. On the TV box, a "headless" mode also stops the launcher and Google services while a model runs.

  3. Experts paged from flash — MoE models only use a few "experts" per token. The project keeps popular experts in RAM, reads others from flash on demand, and prefetches the next layer's experts. This is how 8B and 16B models run on 1 GB of RAM.

  4. Streaming for dense models — For models bigger than RAM, weights stream from flash at the flash's full speed.

How to try it

The quickest start is a ready-made model file from Hugging Face. Important: these files need the patched llama.cpp — stock llama.cpp can't load them.

  • TV box or other ARMv8.2 Android device: see docs/tv-box-setup.md
  • Jailbroken iPad on iOS 12: see ipad-air/README.md
  • All model files and how to rebuild them: see docs/models.md

What's in the repo

Path Contents
patches/ Every change to llama.cpp for the TV box (iPad's patch is in ipad-air/patches/)
tools/ Model rebuilding, GGUF rewriting, benchmarks, kernel checks
tv-llm-ui/ Live token-stream chat page and its launcher
ipad-air/ iOS 12 build, smolchat, deploy and benchmark scripts
bench/ Every chat benchmark (results.jsonl) and raw logs
data/ Vocabulary lists and evaluation texts
tiny-llm-web/ First experiment: a 260K-parameter story model
zigcc/ Zig wrappers for cross-compiling static aarch64 binaries

Documentation

  • docs/tv-box.md — every model run on the TV box, speed and quality
  • ipad-air/README.md — iPad Air results, setup, scripts
  • docs/how-it-works.md — kernels, memory tricks, expert pager, streaming engine
  • docs/models.md — downloads, rebuild targets, checksums
  • docs/lessons.md — what worked and what didn't
  • EXPERIMENT.md — full lab notebook: 160 experiments, failures included

License and credits

  • License: MIT. The llama.cpp patches are derivatives of llama.cpp (MIT). Model weights keep their own licenses; evaluation texts are Wikipedia excerpts (CC BY-SA 4.0).
  • Built on: llama.cpp and llama2.c.
  • Models from: Qwen, unsloth, PrismML (Bonsai), inclusionAI (Ling-mini-2.0), tablinum (Moon-Beam), bartowski, mradermacher, Google (Gemma), and karpathy's stories260K.
  • Other: The iPad is jailbroken with Amethyst; banner uses Archivo font (SIL OFL); 🤏 logo is from Google's Noto Emoji (Apache 2.0).

Key takeaway

smolmaxxing proves that with clever engineering — custom kernels, aggressive memory trimming, and streaming model weights from flash — you can run surprisingly large AI models (even 16B-parameter ones) on devices with only ~1 GB of usable RAM. The techniques are portable: the 1-bit kernel needs only basic NEON instructions, and flash streaming works on any Linux or Android device with more model than RAM.

Related

  • Project
  • Project
  • Project
  • Project