avifenesh/memra

What is memra?

memra is an open‑source inference engine written in Rust that talks directly to NVIDIA GPUs via CUDA. It is tuned for the RTX PRO 6000 (Blackwell) and RTX 5090 work‑station cards and can also be compiled for Hopper H100 cards. The engine loads Hugging Face safetensors checkpoints or GGUF model files and serves them through an OpenAI‑compatible HTTP API (chat‑completion and completion endpoints). It focuses on a single‑model‑per‑GPU deployment (or a two‑GPU pipeline when a model is too large for one card) and provides features such as:

• prefix‑cache reuse and tenant‑scoped admission control to keep VRAM usage predictable,
• MTP speculative decoding (draft‑model acceleration) with automatic fallback,
• support for image and video inputs (the model’s vision tower runs in‑process),
• built‑in authentication, health probes, and metrics.

The project is actively used in production at inference.tiyuvta.ai, where a Qwen 3.8‑27B model is served with the same defaults described in the README.

---

Quick start (run a model locally)

# Install the pre‑built binaries (Linux x86_64, glibc ≥2.35, driver ≥580)
curl -fsSL https://raw.githubusercontent.com/avifenesh/memra/main/tools/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"

# Run a single‑turn chat generation from a local checkpoint or a Hugging Face repo
MEMRA_CHAT=1 run-gen /path/to/hf‑checkpoint \
  --prompt "Explain KV caches in one sentence."

# Start the OpenAI‑compatible server (default bind 127.0.0.1:8080)
MODEL=/path/to/hf‑checkpoint
MEMRA_MODELS="qwen=$MODEL" memra-server

You can then call the API, e.g.:

curl -sS -N http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen","messages":[{"role":"user","content":"Explain KV caches in one sentence."}],"max_tokens":128,"stream":true}'

Authentication, multiple model aliases, and advanced flags are documented in docs/SERVING.md.

---

Performance highlights (RTX PRO 6000 Blackwell)

Metric	Value
Time‑to‑first‑token (cold, p50)	0.156 s (single request)
TTFT (cached turn, 5.7k‑token prefix)	0.130 s
Decode throughput (plain)	140 tokens / s
Speculative‑decoding throughput	240‑245 tokens / s (top‑p/k sampling)
Sustained load	576 requests with 0 errors, 0 sheds
Exactness	8‑byte identical logits to the reference model

On an RTX 5090 laptop the same model runs at ~75 tokens / s plain decode. The README also provides a side‑by‑side table showing 1.1 ×‑2.3 × speedups over llama.cpp for several Qwen models.

---

Supported models & hardware

• Primary targets: RTX PRO 6000 (sm_120a) and RTX 5090 (sm_120a). The engine ships separate pre‑built binaries for these architectures and automatically selects the right code paths at runtime.
• Optional compile‑gated lane: Hopper H100 (sm_90a) – available if you build from source with the appropriate CUDA arch flag.
• Model catalog (selected examples):
  • Qwen 3.8‑27B (dense hybrid, NVFP4+Q5_K quantisation) – served with an MTP draft head.
  • Qwen 3.6‑35B‑A3B (Mixture‑of‑Experts, IQ4_XS) – runs on a single card; larger MoE models can be split across two cards via PP‑2.
  • Gemma‑4 series (MoE or dense, QAT‑quantised) – available with a CLI‑only assistant draft.
  • Ornith‑1.0, Qwen‑AgentWorld, Step‑3.7‑Flash (up to 196 B) – each listed with its quantisation and drafter requirements.

The engine only claims to support the exact model‑quantisation‑drafter combos listed; other checkpoints will be rejected or run with generic (slower) paths.

---

Design boundaries (what memra does not aim to do)

• No tensor‑parallel or multi‑node scaling – it is built for a single workstation GPU (or a two‑GPU PP‑2 split). Large datacenter fleets should look at projects that implement NCCL‑based tensor parallelism.
• Limited model coverage – only the models explicitly measured and tuned are shipped with defaults. Adding a new model requires providing the original safetensors checkpoint (or GGUF) and letting memra repack it into its internal layout.
• No generic CPU fallback – the engine requires a CUDA‑capable NVIDIA GPU; there is no CPU‑only mode.
• Focus on latency‑critical serving – while it can handle high concurrency, the design prioritises low‑latency chat/completion workloads rather than massive batch inference.

---

Where to learn more

• Installation guide – the README’s Install section and tools/install.sh.
• Serving details – docs/SERVING.md (API spec, auth, cache semantics, metrics).
• Performance methodology – docs/PERFORMANCE.md and the raw logs under research/.
• Architecture notes for H100 – ARCHITECTURE-H100.md.
• Flag reference – docs/FLAGS.md.

---

memra is a niche but production‑ready inference stack for developers who own Blackwell‑class RTX GPUs and need a fast, OpenAI‑compatible endpoint without the overhead of larger datacenter‑oriented servers.