smolmaxxing: Running Large Language Models on Low-Resource Hardware
The smolmaxxing project proves that legacy and low-power hardware—such as 13-year-old tablets and budget TV boxes—can effectively run large language models (LLMs) through aggressive optimization of memory, storage, and compute kernels. By treating RAM and flash bandwidth as primary bottlenecks rather than just raw CPU power, the project enables models up to 16B parameters to run on devices with as little as 1 GB of usable memory.
Core Optimization Strategies
The project achieves these results by targeting three critical hardware bottlenecks:
- Compute: Custom kernels for ARMv8.2 architectures (like the Cortex-A55) focus on multiplication-free 1-bit operations, lookup-table kernels for quantized weights, and fused operations for common model architectures like Gated-DeltaNet. These kernels significantly reduce the overhead of in-order CPU execution.
- Memory Management: To fit models within a ~1 GB ceiling, the project employs pruned language model heads, which discard less-frequent vocabulary tokens to save space, and a "headless" mode that suspends non-essential background services on Android devices to maximize available RAM.
- Storage and Streaming: For models exceeding physical RAM, the project implements a sophisticated weight-streaming engine. It uses
O_DIRECTfor efficient flash reads and predictive prefetching to ensure that weights are loaded into memory just before they are needed, effectively turning flash storage into a virtual extension of RAM.
Performance on Legacy Hardware
Performance is measured on two primary test devices, highlighting the efficacy of these techniques:
- Xiaomi TV Box S (3rd Gen): Running an Amlogic S7D processor with 1 GB of usable RAM, the system achieves 7.7 tokens/s on a Qwen3.5-2B model and 5.1 tokens/s on a 16B mixture-of-experts model (Ling-mini-2.0). These results demonstrate that even on modern budget hardware, intelligent memory paging allows for the execution of models far larger than the device's physical RAM.
- iPad Air (2013): Utilizing an Apple A7 chip, the project achieves 23-25 tokens/s on a 1.7B parameter model. By implementing an expert-pager system, the device can even run 16B models at 2.6–4.5 tokens/s, proving that older mobile hardware remains capable of AI inference when software is sufficiently optimized.
Community Insights and Utility
Discussion surrounding the project highlights both the technical achievement and the ongoing debate regarding the utility of "tiny" models. While some users question the practical application of running LLMs on obsolete hardware, others view it as a critical step toward democratizing AI access.
"Models at the intersection of small and useful will be in the toolset that the next generation of poor kids will use to build a more egalitarian world." — Hacker News user nullstyle
Critics point out that raw speed is not the only metric of success. As one user noted, "Speed means nothing if it gets stuck in doom loops, can't tool call, or isn't trained on embedded functionality." The project addresses this by emphasizing the need for models specifically trained for small-scale efficiency, such as those designed for tool-calling or reasoning, rather than simply scaling down larger models which may suffer from degradation at low bit-widths.
Future Directions
The project serves as a practical blueprint for "resurrecting" obsolete technology. By focusing on efficient kernels and memory-mapped weight streaming, developers can repurpose thin clients, routers, and old mobile devices for edge AI tasks. The project's documentation, including a detailed EXPERIMENT.md file, provides a roadmap for others to replicate these optimizations on different ARM-based devices.
Sources
Related
- Project
- Dispatch
- Dispatch
- Dispatch
- Project