Strata 0.1.10 runs Qwen 3.8 Flash Next 125B on RTX 4090 at ~100 tokens/s

TL;DR – What Strata achieves

Strata 0.1.10 makes the 125‑billion‑parameter Qwen 3.8‑Flash‑Next model run on a single consumer GPU (e.g., RTX 4090) at about 100 tokens / second, turning a server‑grade LLM into a desktop‑ready assistant. This matters because it removes the need for costly cloud inference, enabling high‑throughput, privacy‑preserving AI on off‑the‑shelf hardware.

---

Speed that rivals small‑scale models

• On an RTX 4090 (24 GB VRAM) with 128 GB DDR5 RAM, the author reported 124 tokens / s (≈ 100 tokens / s after accounting for overhead).
• Benchmarks on lower‑end cards show the same engine scaling predictably: an RTX 5070 (12 GB) reaches 53 tokens / s for the IQ2_XS quant, while an AMD RX 9070 XT (16 GB) hits 79 tokens / s for the same model.
• The "Writes answers" metric (generation speed) exceeds typical reading speed (≈ 60 tokens / s), meaning the model can output text faster than a human can read it.
• Prompt‑loading speed is also impressive: a 32 K‑token document is ingested at ~1 200 tokens / s on the RTX 5070, enabling near‑real‑time context updates.

"I tried it and it worked surprisingly well. On my machine (Nvidia 4090, 128GB DDR5, Ryzen 7950x3d) I'm getting 124 tokens per sec" – @snehesht (HN comment)

---

Minimum hardware requirements

Component	Minimum	Recommended
GPU	NVIDIA RTX 20/30/40/50 series or AMD Radeon 7900 XT/XTX, 7800 XT, 7700 XT, 9060 XT, 9070/9070 XT, 6800/6900 series	24 GB VRAM (RTX 4090/3090) for best speed; 12 GB VRAM works with reduced throughput
System RAM	32 GB (fits the smallest "Coder" model)	64 GB+ for full model suite (IQ2_XS, IQ3_XXS, IQ3_S)
Disk	~80 GB free, SSD preferred for fast model loading	NVMe SSD for optimal swap performance
OS	Windows 10/11 or Linux with up‑to‑date drivers	—

The installer automatically detects your hardware and selects the largest model that fits.

---

One‑click installation (or AI‑assisted setup)

1. Download the repository (git clone https://github.com/Niko1221/Strata.git).
2. Run START-HERE.bat on Windows or ./setup.sh on Linux. The script:
   • Detects GPU, RAM, and disk space.
   • Prompts for model size, context length, and image support (press Enter for defaults).
   • Downloads the ~70 GB GGUF model file and extracts it.
3. Wait 1‑3 minutes while the model loads into RAM and the GPU cache is populated. The window shows progress; the PC may feel sluggish during this phase.
4. Open http://127.0.0.1:8080 in a browser to access the Strata UI (Chat, Monitor, About).

AI‑assisted installation

If you prefer a language‑model to drive the setup, paste the following into your coding assistant:

Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.

The assistant will run the same installer steps automatically.

---

Choosing the right quantized model

RAM	Recommended model	Reason
32 GB	Coder (32 GB)	Optimized for code; fits tight memory
48 GB	IQ2_XS (or Q2_0)	Smallest that fits, fastest generation
64 GB	IQ2_XS (default) or IQ3_XXS / IQ3_S	All sizes fit; IQ3_S offers highest quality at modest speed loss
96 GB+	IQ3_S or Unsloth UD‑IQ4_XS (~4‑bit)	Largest experts loaded; UD‑IQ4_XS trades speed for lower VRAM usage

• Coder: 91 % of full‑model SWE‑bench score, but weaker on non‑code languages.
• IQ2_XS: 2‑bit quant, best speed‑to‑quality ratio for most tasks.
• IQ3_S: 3‑bit quant, highest quality among the supported sizes; runs at ~53 tokens / s on RTX 5070.
• Unsloth UD‑IQ4_XS: ~4‑bit, 94 GB download; reads part of the model from SSD, resulting in slower generation on low‑RAM systems.

---

Using the model in your own apps

• OpenAI‑compatible endpoint: http://127.0.0.1:8080/v1. Any client that accepts an OpenAI API key can point to this URL; the key can be any string.
• Anthropic‑compatible endpoint: http://127.0.0.1:8080/v1/messages (set ANTHROPIC_BASE_URL accordingly).
• Batching: By default Strata processes one request at a time. Set "parallel": 2 in config.json to allow two concurrent streams (may reduce per‑request speed on 12 GB cards).
• Image input: Enable during installation; on Windows AMD cards currently lack image support.
• Remote access: Run START-HERE.bat --setup --host 0.0.0.0 --api-key <secret> to expose the API on the LAN (always protect with a secret key).

---

Community feedback and common questions

"We're getting closer and closer to the day we can have an Opus‑like model running locally. The dream!" – @SuperV1234 (HN comment)

"The model is a team of 24,576 small specialists ('experts')…" – Strata documentation, explaining the MoE off‑load strategy.

"I am seeing > 110 tokens/sec on a RTX 4090 with 128 GB DDR4 RAM" – @mrinterweb (HN comment)

"Getting about 200 tok/s on a 5090 with 64 GB RAM, running the swift iq2_xs quant" – @IronWolve (HN comment)

Common concerns

• Quality vs. speed: Users report a modest drop in benchmark scores when moving from IQ2_XS to IQ3_S, but the trade‑off is often worth the higher fidelity for complex coding or multilingual tasks.
• GPU memory limits: Cards with < 12 GB VRAM cannot run any of the supported quantizations; the community is experimenting with older GPUs (Tesla V100, GTX 10 series) using experimental patches, but performance is dramatically lower.
• Parallel requests: Strata currently queues requests; true parallel inference requires enough VRAM to hold separate expert subsets for each stream.
• Image support on AMD/Windows: Not yet available; Linux users can process images via the CPU fallback.

---

How Strata makes a 125 B model fit on 12‑24 GB VRAM

1. Mixture‑of‑Experts (MoE) off‑loading – The model consists of 24,576 experts; only a few thousand active experts are cached on the GPU at any time.
2. RAM‑resident expert pool – All experts reside in system RAM; the CPU swaps in the needed experts while the GPU handles the hot subset.
3. SSD lookup table – A large static table stored on SSD speeds up expert selection, reducing CPU‑RAM traffic.
4. Guess‑and‑check pipeline – A lightweight helper proposes candidate tokens; the full MoE validates them, yielding a 1.6‑1.8× speedup over naïve decoding.
5. Chunked context processing – Long prompts are read in 8 192‑token blocks at > 1 000 tokens / s, enabling 32 K‑token context windows without stalling the GPU.

The full technical description is in the repository’s HOW_IT_WORKS.md and the accompanying paper (Strata-Paper.pdf).

---

Licensing and credits

• Model: Qwen 3.8‑Flash‑Next (MIT‑compatible usage via Hugging Face).
• Quantization: Performed by ISTA‑DASLab, UkisAI (Swift 1.5), and Unsloth.
• Engine: Built on llama.cpp / ggml (BSD‑3‑Clause).
• Strata code: MIT License (see LICENSE in the repo).
• Acknowledgements: The Qwen team, the compression contributors, and the open‑source community that maintains llama.cpp.

---

Getting started quickly

# Clone and run on Linux
git clone https://github.com/Niko1221/Strata.git && cd Strata
./setup.sh   # follows prompts, downloads ~70 GB model
# Open the UI
xdg-open http://127.0.0.1:8080

On Windows, double‑click START-HERE.bat after extracting the zip.

---

Bottom line

Strata demonstrates that a 125‑billion‑parameter LLM can be run locally on consumer‑grade GPUs with token generation speeds that make interactive use practical. By off‑loading experts to RAM and SSD, it sidesteps the traditional VRAM ceiling, opening the door for privacy‑first, low‑latency AI applications without relying on expensive cloud inference.