Niko1221/Strata

Strata – Run a 125‑billion‑parameter LLM on a consumer gaming PC

What it is

• An open‑source engine that lets you run the Qwen‑3.8‑Flash‑Next large language model (≈125 B parameters) locally on a desktop with a modern GPU (12 GB VRAM +).
• The model is heavily compressed (GGUF formats like Q2_0, IQ2_XS, IQ3_S, etc.) so it can fit into RAM/VRAM and be streamed from SSD when needed.
• Provides an OpenAI‑compatible HTTP API (http://127.0.0.1:8080/v1) and a tiny web UI for chat, monitoring, and image input.

Why it matters

• Large‑scale LLMs normally require server‑grade GPUs with 40‑100 GB memory. Strata’s “expert‑splitting” technique spreads the model across GPU, system RAM, CPU and SSD, making a 125 B model usable on a typical gaming rig.
• Keeps all data on‑device, which is good for privacy and offline work.

Key features

Feature	Details
Supported hardware	NVIDIA RTX 20‑50 series, AMD RX 7900 XT/XTX, RX 7800 XT, RX 9070 XT, etc. Minimum 12 GB VRAM, 32 GB RAM (64 GB recommended).
Model sizes	Multiple GGUF quantizations: Q2_0, IQ2_XS, IQ3_XXS, IQ3_S, Coder, Swift 1.5, Unsloth UD‑IQ4_XS, etc. Smaller files run faster; larger ones are more capable.
Performance	On an RTX 5070 (12 GB) the fastest Q2_0 runs ~94 tokens/s for generation and ~2 650 tokens/s for prompt ingestion; IQ3_S gives ~53 tokens/s. Benchmarks for AMD cards are also listed.
Installation	One‑click scripts (START-HERE.bat / setup.sh). The installer detects GPU, picks a suitable model, downloads ~70 GB of weights, and launches a local server.
API	OpenAI‑compatible /v1/chat/completions, /v1/completions, /v1/responses, plus Anthropic‑style /v1/messages. Works with any client (Claude Code, Cursor, Copilot, etc.).
Image support	Optional picture input on AMD Linux builds; Windows builds currently lack it.
Multi‑GPU	Experimental support for sharing the model across two or three cards.
Parallelism	Single‑request mode by default; can enable limited batching ("parallel": 2).
Monitoring	Web UI shows GPU/CPU/RAM usage, token throughput, and a live “Monitor” tab.
Extensibility	Add new GGUF models via SETUP.bat/setup.sh --setup.

How it works (high‑level)

• The 125 B model is split into 24 576 “experts”. The most frequently used experts stay on the GPU; the rest reside in system RAM, while a lookup table lives on the SSD.
• A lightweight “guess” module predicts candidate tokens, then the full model verifies them, giving a 1.6‑1.8× speed‑up.
• Prompt ingestion is chunked (up to 8 192 tokens at a time) at >1 000 tokens/s, enabling 32 KB‑level context windows.

Getting started

1. Ensure you have a supported GPU, ≥32 GB RAM, and ~80 GB free disk space.
2. Clone the repo or download the zip, then run START-HERE.bat (Windows) or ./setup.sh (Linux).
3. Follow the prompts – the installer picks a model size that fits your RAM and downloads it.
4. Open http://127.0.0.1:8080 in a browser to chat or point your development tools at http://127.0.0.1:8080/v1.

Typical use cases

• Local coding assistant that can read large codebases (up to 32 KB context) without sending data to the cloud.
• Offline research or creative writing where privacy matters.
• Experimentation with LLM inference on consumer hardware for education or hobby projects.

Limitations

• Requires a fairly powerful PC (12 GB VRAM, 32‑64 GB RAM).
• First launch can freeze the system for a few minutes while the model loads into RAM/VRAM.
• Only one request at a time by default; higher parallelism reduces per‑request speed.
• Image input works only on AMD Linux builds at the moment.

License & community

• Strata’s code is MIT‑licensed.
• The underlying model weights are under the licenses of Qwen 3.8‑Flash‑Next and the various GGUF compressors (ISTA‑DASLab, UkisAI, Unsloth).
• Contributions are welcomed; benchmarks and troubleshooting tips are collected in the docs/COMMUNITY_BENCHMARKS.md file.

Where to learn more

• Detailed speed tables: docs/MODELS.md and docs/DETAILS.md.
• Installation guide: docs/INSTALL.md.
• Technical deep‑dive: docs/HOW_IT_WORKS.md and the accompanying paper (docs/paper/Strata-Paper.pdf).

---

Strata lets a consumer‑grade gaming PC act like a small AI server, giving you private, offline access to a 125 B‑parameter LLM with a familiar OpenAI‑style API.