mira-wm/mira

Code for MIRA: Multiplayer Interactive World Models with Representation Autoencoders

What it solves

MIRA addresses the challenge of creating real-time, interactive world models for complex multiplayer environments. It allows for the simulation of a full 2v2 Rocket League match, generating video frames based on the actions of all four players simultaneously.

How it works

The system uses a 5-billion parameter latent diffusion model that generates video frames sequentially. It leverages a representation autoencoder (codec) to handle the visual data. The training process involves a codec, a single-player world model, and finally a 4-player world model that is warm-started from the single-player version. It is trained on the "Rocket Science" dataset, which contains synchronized views, keyboard actions, and game states from 2v2 matches.

Who it’s for

Researchers and developers working on world models, latent diffusion for video generation, and AI for complex interactive simulations.

Highlights

  • Real-time performance: Capable of running a full 2v2 match at 20 FPS on a single GPU.
  • Multiplayer interaction: Generates frames based on the combined actions of four players.
  • Large-scale model: Utilizes a 5B parameter latent diffusion architecture.
  • Comprehensive dataset: Includes the "Rocket Science" dataset with synchronized multi-view video, actions, and physics state.

Related

  • Project
    nv-tlabs/Gamma-World

    Gamma-World is a generative multi-agent world model that simulates shared environments for multiple independently controllable agents, supporting real-time interactive video generation at 24 FPS.

  • Project
    EternalEvan/Astra

    Astra is an interactive world model that uses an autoregressive diffusion transformer to generate realistic, action-conditioned long-horizon video predictions.

  • Project
    Tencent-Hunyuan/HY-WorldPlay

    HY-World 1.5 is a real-time interactive world modeling framework that uses streaming video diffusion to generate geometrically consistent 3D environments based on user input.

  • Dispatch
    SANA-WM: Breaking the Minute Barrier in Open-Source World Modeling

    NVIDIA researchers introduce SANA-WM, a 2.6B parameter model capable of generating high-fidelity, 720p, minute-long videos with precise camera control on a single GPU.

  • Project
    xandergos/terrain-diffusion-mc

    A Minecraft Fabric mod that uses a diffusion model to generate highly diverse and realistic terrain and landscapes.