flashrt-project/FlashRT

FlashRT – a real‑time inference engine for low‑batch, latency‑critical AI

What it is – FlashRT is a C++/CUDA library that bundles hand‑written kernels (norm, activation, fused residual‑norm‑quant, RoPE, FP8/NVFP4 GEMM, Flash‑Attention, etc.) into a static CUDA‑graph that can be replayed with virtually no Python overhead. It targets workloads where a single request must be answered as fast as possible (robotic control, vision‑language models, video generation, long‑context LLM serving), rather than the high‑throughput batch‑oriented serving stacks like TensorRT, vLLM or SGLang.

Key ideas

• No compilation step – models are loaded directly from safetensors/Orbax and the first call captures the whole forward pass into a static graph. Subsequent calls are just graph replays (≈3 s warm‑up, then sub‑millisecond Python cost).
• Hardware‑agnostic kernel library – the same kernel binaries run on Jetson AGX Thor (edge) up to RTX 5090/4090/A100 (server). NVIDIA‑specific implementations are provided out of the box.
• FlashRT Structures – a thin plug‑in layer that can attach the kernel catalog to an unmodified PyTorch, JAX, or Hugging‑Face model. No forking or source edits are required; the host model stays unchanged while the runtime routes tensors through the fast kernels.
• FP8 / NVFP4 support – automatic per‑tensor calibration (cached as JSON) lets 8‑bit floating‑point formats be used for both weights and activations, dramatically cutting memory and compute while preserving >0.999 cosine similarity to the FP16/BF16 baseline.
• Unified serving APIs – a three‑line Python interface (flash_rt.load_model(...).predict(...)) works for image‑text‑video generation, VLA robot control, and OpenAI‑compatible LLM/audio endpoints.

What it can do (as shown in the repo)

Domain	Model	Hardware	Latency / Throughput
VLA robot control	Pi0.5, GROOT N1.6/N1.7	Jetson AGX Thor, RTX 5090	8 ms → 29 ms per decision (3.8 Hz → 34 Hz)
Vision‑language	Qwen‑3‑VL‑8B	RTX 5090	65 → 146 tokens / s (≈1.8× speed‑up)
LLM (Mixture‑of‑Experts)	Qwen‑3.6‑35B‑A3B (NVFP4)	RTX 5090	52 → 285 tokens / s, 256 K context support
Video generation	Wan2.2‑TI2V‑5B	RTX 5090	6.48 s → 1.68 s total (≈4× faster)

Why it matters – Many real‑world AI systems (autonomous robots, interactive assistants, on‑device video tools) cannot afford the latency of batch‑oriented servers. FlashRT’s static‑graph replay and low‑precision kernels give sub‑30 ms end‑to‑end response on edge GPUs, while still scaling to large LLMs on server‑class GPUs.

Getting started

from flash_rt import structures

# Load any Hugging‑Face / PyTorch model (no code changes needed)
model = flash_rt.load_model("Qwen3.6-35B-A3B", precision="nvfp4")

# Attach the fast kernel pipeline
plan = structures.attach(model, model.forward)
print(structures.explain(plan))   # shows which ops are kept, fused, or rejected

# Simple inference loop
loop = structures.decode_loop(model, max_len=4096)
output = loop.generate(input_ids, max_new_tokens=256)

The repository also ships ready‑to‑run examples (examples/structure_pipeline/), benchmark scripts, and a set of serving containers that expose OpenAI‑compatible HTTP endpoints for LLM or audio generation.

Where it fits in the ecosystem

• Complementary to TensorRT (which excels at high‑batch, compiled engines) and to vLLM/SGLang (which excel at massive concurrent LLM serving). FlashRT fills the niche of single‑stream, low‑batch, real‑time inference.
• Integrates with existing Python ecosystems (PyTorch, JAX, Hugging‑Face Transformers, Diffusers) via the Structures plug‑in, so you can keep your familiar training code and just swap the runtime.
• Open‑source – the core engine lives in flashrt-project/FlashRT; related kernel packs are in FlashRT-HF-kernels, and the C++ runtime for embedded robots is in FlashRT-Nexus.

Documentation & community

• Paper: https://arxiv.org/abs/2606.20537
• Kernel hub: https://github.com/flashrt-project/FlashRT-HF-kernels and https://huggingface.co/flashrt
• Demo videos and walkthroughs: https://github.com/flashrt-project/FlashRT-assets and the Hugging‑Face Space linked in the README.
• Active changelog (July‑Aug 2026) showing rapid addition of new models (Qwen‑3‑VL, Cosmos3‑Edge, Higgs Audio, OmniVoice) and hardware support (Jetson Orin, RTX 5090).

---

Bottom line – FlashRT is a genuine, high‑performance inference engine focused on latency‑critical, small‑batch AI workloads. It provides hand‑tuned CUDA kernels, static‑graph replay, and a plug‑in system that lets you accelerate existing PyTorch/JAX models without rewriting them, making it especially useful for robotics, real‑time vision‑language agents, and low‑latency LLM serving on both edge and server GPUs.