Waypoint-1: Real-time Interactive Video Diffusion from Overworld

Overworld has introduced Waypoint-1, a real-time interactive video diffusion model that enables users to step into and interact with generated worlds using text prompts, mouse movements, and keyboard inputs. Unlike traditional world models that fine-tune pre-trained video models with limited controls, Waypoint-1 is designed from the ground up for interactive experiences, providing zero-latency control and the ability to run on consumer hardware.

Model Architecture and Training

Waypoint-1 is a latent model based on a frame-causal rectified flow transformer. It was trained on a dataset of 10,000 hours of diverse video game footage paired with text captions and control inputs.

Training Methodology

To achieve its interactive capabilities, Waypoint-1 utilized two primary training stages:

  1. Diffusion Forcing: The model was pre-trained using diffusion forcing, where it learns to denoise future frames given past frames. A causal attention mask ensures tokens in a frame only attend to their own frame or previous frames, allowing for the procedural generation of new frames during inference.
  2. Self-Forcing: To resolve the inference mismatch and error accumulation (noisy long rollouts) caused by random noising during pre-training, the model underwent post-training with self-forcing via DMD. This technique aligns the training regime with inference behavior, enabling one-pass CFG and few-step denoising.

WorldEngine Inference Library

Overworld released WorldEngine, a high-performance Python inference library designed for low-latency, high-throughput streaming of interactive world models. WorldEngine consumes context frames, text, and keyboard/mouse inputs to output image frames in real-time.

Performance Benchmarks

Running the Waypoint-1-Small (2.3B parameter) model on an NVIDIA RTX 5090, WorldEngine achieves the following performance:

  • Throughput: Approximately 30,000 token-passes per second (based on a single denoising pass of 256 tokens per frame).
  • Frame Rate: 30 FPS at 4 denoising steps, or 60 FPS at 2 denoising steps.

Technical Optimizations

WorldEngine's performance is driven by four specific optimizations:

  • AdaLN Feature Caching: Caches and reuses AdaLN conditioning projections when prompt conditioning and timesteps remain constant between forward passes.
  • Static Rolling KV Cache + Flex Attention: Implements advanced caching and attention mechanisms to reduce redundant computation.
  • Matmul Fusion: Uses fused QKV projections to optimize matrix multiplication.
  • Torch Compile: Utilizes torch.compile(fullgraph=True, mode="max-autotune", dynamic=False) for maximum graph optimization.

Availability and Implementation

Waypoint-1 weights are available on the Hugging Face Hub, with the Waypoint-1-Small model currently available and Waypoint-1-Medium coming soon. Users can experience the model via Overworld Stream.

Implementation Example

Developers can integrate Waypoint-1 using the following WorldEngine pattern:

from world_engine import WorldEngine, CtrlInput

# Create inference engine
engine = WorldEngine("Overworld/Waypoint-1-Small", device="cuda")

# Specify a prompt
engine.set_prompt("A game where you herd goats in a beautiful valley")

# Optional: Force the next frame to be a specific image
img = pipeline.append_frame(uint8_img)  # (H, W, 3)

# Generate 3 video frames conditioned on controller inputs
for controller_input in [
        CtrlInput(button={48, 42}, mouse=[0.4, 0.3]),
        CtrlInput(mouse=[0.1, 0.2]),
        CtrlInput(button={95, 32, 105}),
]:
    img = engine.gen_frame(ctrl=controller_input)

Sources