rockmatrix-ai/EdgeRock

EdgeRock – A 4‑Billion‑parameter multimodal model that reasons with constant memory

What it is

• EdgeRock‑4B is an open‑source language model (text + image) that can handle very long contexts while keeping its memory usage fixed, regardless of how many tokens have been processed.
• It achieves this by combining three ideas:
  1. Recurrent state – a compact matrix that continuously summarizes everything seen so far.
  2. Sliding‑window attention – normal attention limited to the most recent ~1 k tokens.
  3. Rock Memory Anchors (RMA) – a fixed‑size bank (1 024 records) that stores selected “important” key‑value pairs from the distant past and can be queried later.

Why it matters

• Traditional transformers need a KV cache that grows linearly with sequence length, quickly exhausting GPU memory on long‑context tasks.
• EdgeRock’s O(1) memory footprint lets it run on a single 80 GB Hopper GPU even with 256 k‑token inputs, making it suitable for edge devices that must share memory with perception or control modules.

Key performance numbers (from the repo’s technical report)

Metric	EdgeRock‑4B	Typical full‑attention baseline (Qwen3.5‑4B)
Mean pass@1 on AIME 2024‑2026	82.9 %	86.1 % (full‑attention)
Mean accuracy on 29 text benchmarks	77.2 %	71.4 % (Qwen3.5‑4B)
Mean visual‑benchmark accuracy	84.9 %	89.1 %
Attention KV memory per sequence	72 MiB (constant)	8 GiB at 256 k tokens (grows with length)
Decode throughput (128 k‑in + 128 k‑out, batch 1)	233.7 tokens/s	151.3 tokens/s
Best‑batch output‑token throughput (32 k‑in + 32 k‑out)	10,669 tokens/s (batch 512)	1,889 tokens/s (batch 64)

How to try it

# Clone and set up a virtual environment
git clone https://github.com/rockmatrix-ai/EdgeRock.git
cd EdgeRock
python -m venv .venv && source .venv/bin/activate
python -m pip install ./wheels/transformers-5.7.0.dev0-py3-none-any.whl -e .

# Download the 4B checkpoint from Hugging Face
hf download stonehill-tech/EdgeRock --local-dir checkpoints/EdgeRock-4B \
  --include "*.safetensors" "*.json" "*.jinja" "tokenizer*" "vocab*" "merges*" \
  --exclude "evaluation_data/*"

# Run a quick demo (the repo ships a tiny wrapper script)
python run.py demo \
  --model checkpoints/EdgeRock-4B \
  --prompt "Solve 17 × 23, explaining your reasoning." \
  --max-new-tokens 2048

• The provided backend works on CPU and CUDA. For the fastest speed on Nvidia Hopper GPUs, follow the extra steps described in USAGE.md (CUDA‑graph + fused BF16 ops).

Architecture at a glance

• Built on the Qwen3.5‑4B backbone, but replaces the eight full‑attention layers with a pattern:

  [GDN, GDN, GDN, sliding‑window attention ∥ RMA] × 8
  

• Persistent history consists of:
  • Recurrent matrices (FP32) – fixed‑dimensional summary.
  • Local‑window KV (31.97 MiB per sequence).
  • RMA anchor bank (32 MiB) + pending candidates (8 MiB).
• All components have a hard capacity, so memory stays O(1) with respect to token length.

Evaluation highlights

• Mathematical reasoning: 97.5 % on MATH‑500, 82.9 % mean pass@1 on AIME, beating bounded‑state baselines by >60 pp.
• Coding: 61.6 % on LiveCodeBench v6 (still behind the full‑attention Qwen3.5‑4B but competitive for its size).
• Visual tasks: 84.9 % mean accuracy across six benchmarks, retaining >95 % of the backbone’s performance.
• Distant‑memory queries: 94.1 % correct when RMA is enabled, versus 0.4 % when disabled, showing the anchors are essential for recalling far‑away information.

Who might use EdgeRock

• Researchers exploring long‑context reasoning without the memory blow‑up of classic transformers.
• Developers building on‑device AI assistants that need to keep a running “state” while also processing images.
• Anyone needing high‑throughput inference on very long prompts (e.g., document‑level QA, code‑base analysis) on a single GPU.

Where to find more

• Technical report (PDF) – deep dive into the design and experiments.
• Model weights on Hugging Face: stonehill-tech/EdgeRock.
• Full benchmark tables and inference guide are in the repo (USAGE.md).

---

All numbers are taken directly from the repository’s README and technical report; no external claims are added.