invergent-ai/surogate

Surogate – High‑performance LLM training & serving engine (C++/CUDA)

What it is – Surogate is a native C++/CUDA toolkit that lets you train large language models (including LoRA/QLoRA, reinforcement‑learning methods, distillation, MoE, and multimodal fine‑tuning) and serve the resulting checkpoints with an OpenAI‑compatible HTTP server. The project focuses on squeezing the most tokens per second out of NVIDIA GPUs by using compiled training graphs, fused kernels, low‑precision tensor‑core formats (BF16, FP8, NVFP4, GGUF), and aggressive memory‑management tricks.

---

Core capabilities

Area	Highlights
Training engine	• Full pre‑training, SFT, LoRA/QLoRA, GRPO (RL), DPO, knowledge distillation. • Precision recipes: BF16, hybrid FP8, Blackwell‑NVFP4, plus quantized adapters. • Multi‑GPU & multi‑node support (threaded data‑parallel, ZeRO sharding, Ray integration). • Pipeline parallelism & CPU offload for models larger than a single GPU. • MoE expert parallelism, load‑balancing, and imbalance detection. • Automatic graph compilation and ahead‑of‑time autodiff for low overhead.
Serving engine	• OpenAI‑compatible Chat/Completions APIs (also Anthropic‑style messages). • Streaming output, tool‑call parsing, “thinking” controls.
• Continuous batching, prefix caching, up to 128 concurrent sequences per model.
• Speculative decoding (MTP/DFlash) and multi‑GPU layer pipelines.
• Native GGUF loading (Q4_K_M, Q8_0, etc.) and direct safetensors import.
• Runtime LoRA loading/unloading, multi‑model hosting, sleep‑/wake‑up of idle models.
• Vision & embedding endpoints (image/video, EmbeddingGemma on GPU/AVX‑512).
• Prometheus metrics, API‑key auth, health checks.

---

Performance claims (as of Sep 2026)

Task	Hardware	Tokens / s	Relative gain
Training – Qwen3‑0.6B BF16	1 × H100	53,900	2.53× vs. Unsloth (1 × H100)
	4 × RTX 5090	136,200 (FP4 LoRA)	3.74× aggregate vs. single‑card
Serving – Qwen3.5‑0.8B (1 user)	1 × RTX 5090	802	2.32× vs. vLLM
	8 × RTX 5090, GLM‑5.3‑Flash (16 users)	292.9	7.0× vs. llama.cpp
	100 users, Qwen3.5‑4B	5,345	1.19× vs. vLLM

Benchmarks are measured on packed 2,048‑token sequences (training) or 512‑input / 128‑output workloads (serving) with wall‑time token counts; model loading time is excluded.

Getting started (Linux, Python 3.12, CUDA 12.8+)

# Install the pre‑built wheel (selects the right CUDA build)
curl -LsSf https://github.com/invergent-ai/surogate/releases/latest/download/install.sh | bash
source .venv/bin/activate

Serve a model

surogate serve Qwen/Qwen3.5-0.8B \
  --served-model-name surogate \
  --port 8080 --max-model-len 4096 \
  --max-num-seqs 16 --kv-capacity auto

Train a LoRA adapter (example train.yaml shown in the README) then run:

surogate sft train.yaml

The CLI also provides merge, quantize, grpo-colocate, and Docker images (ghcr.io/invergent-ai/surogate:latest-cu129).

Who might use it

• Research labs / startups that need to iterate quickly on fine‑tuning or RLHF pipelines and want deterministic, low‑latency inference on the same codebase.
• Enterprises deploying many concurrent LLM endpoints on RTX 50‑series or H100 clusters and looking for higher throughput than vLLM/llama.cpp.
• Developers of custom model families (Qwen, Llama 3, Gemma 4, MiniCPM5, etc.) who want a single native engine for both training and serving, with built‑in support for LoRA, MoE, and vision extensions.

---

Limitations / open items

• The serving build targets SM120a (Blackwell/RTX 50) by default; Ada/Hopper GPUs run a fallback build that may lack some features.
• Certain model families (e.g., DeepSeek‑V4, Flash‑Next, GLM‑5.3‑Flash) have training definitions marked “deferred” – they can be served but not yet trained.
• Runtime LoRA is not yet supported for the Spark‑X2.5 serving path.
• GPU‑only inference; CPU‑only generation is not provided.

---

Where to learn more

• Docs – https://docs.surogate.ai (installation, training‑mode guide, precision recipes, RL guides, serving API reference).
• Benchmarks – docs/reference/benchmarks.md and surogate/serve/BENCHMARKS.md.
• Examples – examples/ directory contains end‑to‑end scripts for SFT, MoE, vision, GRPO, DPO, and quantization.
• Source – C++/CUDA core lives under csrc/; the Python DSL and CLI are in surogate/.

---

TL;DR

Surogate is a performance‑first, native C++/CUDA platform that unifies LLM training (including LoRA, RL, MoE, and multimodal fine‑tuning) and high‑throughput serving behind OpenAI‑compatible APIs. It leverages compiled graphs, fused kernels, and low‑precision formats to claim 2‑7× speedups over popular Python‑based stacks on modern NVIDIA GPUs.