NVIDIA/flashdreams

high-performance inference and serving library for interactive autoregressive video and world models

FlashDreams – High‑Performance Real‑Time Inference for Video & World Models

What it is – FlashDreams is an open‑source library (Apache‑2.0) that lets you run large autoregressive video‑generation and world‑model networks at interactive speeds. It started as the runtime behind NVIDIA’s OmniDreams demo and has been generalized into a platform for any “real‑time” generative model used in games, autonomous‑vehicle simulation, robotics, or virtual environments.

Key capabilities

  • Real‑time serving – Optimized GPU kernels, multi‑GPU scheduling and a low‑latency pipeline so that a model can generate video frames or world‑state updates faster than they are displayed.
  • Model catalogue – First‑party integrations for a range of NVIDIA‑released models (Self‑Forcing, OmniDreams, LingBot‑World, Waypoint 1.5, Wan2.1, Causal‑Forcing, FlashVSR, Cosmos‑Predict2.5, etc.). Each integration ships with a ready‑to‑run runner, benchmark numbers and multi‑GPU launch scripts.
  • Flexible deployment – Install via pip or use the uv‑based workflow that pulls the correct CUDA‑13 (or optional CUDA‑12) PyTorch stack. A Docker image with the NVIDIA Container Toolkit is also supported.
  • Extensible architecture – The repo provides an ARCHITECTURE.md, a config system, and a “new integration” guide so developers can plug in their own autoregressive video or world‑model code.
  • Interactive demos – Includes a driving demo (OmniDreams) that lets a user control a world model through keyboard/mouse or WebRTC, showcasing closed‑loop generation.

Typical workflow

  1. Install the library (uv sync --extra runners or pip install flashdreams).
  2. Pull a model checkpoint from Hugging Face (requires an HF token).
  3. Launch a runner, e.g. flashdreams-run-v2 t2v-self-forcing‑wan2.1‑t2v‑1.3b … --prompt "A cat surfing.".
  4. The runner streams generated frames to a video file or an interactive window.

Hardware requirements – Designed for NVIDIA GPUs with ≥ 80 GB VRAM (H100 80 GB is the reference). Consumer GPUs will usually run out of memory; the README lists per‑model VRAM needs.

Who should use it – Researchers building or evaluating high‑throughput video generation, engineers needing a production‑grade inference server for world‑model‑based simulation, and developers who want a plug‑and‑play way to run NVIDIA’s latest generative video models in real time.

Where to learn more – Full docs at https://nvidia.github.io/flashdreams, model‑specific pages, and the NVIDIA Omniverse Discord channel #flashdreams for community support.

Related

  • Project
    nv-tlabs/omni-dreams

    A world model that generates real-time, photorealistic multi-camera video for autonomous driving simulation using HD maps and trajectory poses.

  • Project
    NVIDIA/TensorRT-LLM

    TensorRT‑LLM is NVIDIA’s open‑source library that compiles large language (and visual‑generation) models into ultra‑fast TensorRT inference engines. It provides custom CUDA kernels, quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token throughput while keeping accuracy. The Python‑friendly runtime integrates with Triton, Kubernetes, and Jetson, and includes docs and examples for models like GPT‑OSS‑120B, Llama‑4, DeepSeek‑R1, and diffusion models.

  • Project
    InfiniTensor/InfiniTensor

    InfiniTensor is an open‑source C++ runtime that imports ONNX graphs, applies generic graph rewrites, and executes them on CPU or on custom accelerator back‑ends (InfiniOps/InfiniRT). It offers Python bindings, a test suite, and a modular design that lets hardware vendors plug in their own kernels.

  • Project
    flashrt-project/FlashRT

    A high-performance real-time inference engine optimized for small-batch, latency-sensitive AI workloads across NVIDIA, AMD, and Ascend hardware.

  • Project
    openvinotoolkit/openvino

    An open-source toolkit for optimizing and deploying deep learning models across CPU, GPU, and NPU hardware to boost inference performance.