flashrt-project/FlashRT

FlashRT is a high-performance realtime inference engine for small-batch, latency-sensitive AI workloads. The flagship integration is production VLA control for Pi0, Pi0.5, GROOT N1.6, and Pi0-FAST. Also support llm e.g, qwen3.6-27B

What it solves

FlashRT is designed for small-batch, latency-sensitive AI workloads where real-time response is critical. It solves the performance bottlenecks of standard inference frameworks by providing a high-performance engine specifically tuned for low-latency execution across diverse hardware, including NVIDIA GPUs, AMD GPUs, and Ascend NPUs.

How it works

FlashRT achieves its speed through several optimization techniques:

  • Hand-written Kernels: Highly tuned kernels for attention, GEMM, normalization, and quantization tailored for specific hardware backends (CUDA, HIP, AscendC).
  • Static Execution: Uses static graph execution (CUDA/HIP/NPU graphs) over stable buffers to eliminate runtime overhead.
  • Quantization: Implements production-ready FP8 (E4M3) and NVFP4 with automatic per-tensor calibration.
  • Cross-Layer Fusion: Combines multiple operations to reduce memory access and compute overhead.
  • Direct Loading: Loads safetensors or Orbax checkpoints directly without requiring a separate compilation or export step.
  • FlashRT Structures: A plugin system that allows the kernel catalog to be attached to unmodified PyTorch hosts (like transformers or diffusers) without modifying the host's source code.

Who it’s for

It is primarily for developers and researchers working on:

  • Robotics: Specifically VLA (Vision-Language-Action) control for models like Pi0, GROOT, and LingBot.
  • Real-time AI Agents: Applications requiring single-stream LLM or VLM inference with minimal latency.
  • Edge Deployment: Users deploying on hardware like Jetson AGX Thor or RTX 5090.
  • Audio/Video Generation: Real-time TTS (Higgs Audio) or video-policy paths (Wan2.2).

Highlights

  • Multi-Hardware Support: Native backends for NVIDIA, AMD (including MI300X/MI350X), and Ascend NPUs.
  • Extreme Low Latency: Significant speedups over torch.compile and eager mode, particularly for VLA models on Jetson Thor.
  • Seamless Integration: Can be integrated into existing PyTorch/JAX pipelines via a 3-line API or the flashrt-structures package.
  • Broad Model Support: Optimized for a wide range of models including Qwen3 (LLM/VL), Pi0, and GROOT.

Related

  • Project
    NVIDIA/TensorRT-LLM

    TensorRT‑LLM is NVIDIA’s open‑source library that compiles large language (and visual‑generation) models into ultra‑fast TensorRT inference engines. It provides custom CUDA kernels, quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token throughput while keeping accuracy. The Python‑friendly runtime integrates with Triton, Kubernetes, and Jetson, and includes docs and examples for models like GPT‑OSS‑120B, Llama‑4, DeepSeek‑R1, and diffusion models.

  • Project
    NVIDIA/TensorRT

    A high-performance inference optimizer and runtime from NVIDIA that accelerates AI model deployment and throughput on NVIDIA GPUs.

  • Project
    tile-ai/TileRT

    A tile-based runtime engine for ultra-low-latency LLM inference that optimizes hardware utilization to achieve millisecond-level time per output token.

  • Project
    alibaba/rtp-llm

    A high-performance LLM inference acceleration engine developed by Alibaba that optimizes throughput and latency for production-scale deployments.

  • Project
    Avarok-Cybersecurity/atlas

    A pure Rust and CUDA LLM inference engine that provides a lightweight, high-throughput server for large models without requiring Python or PyTorch.