commaai/commavq

A dataset of 100,000 minutes of driving video compressed using a VQ-VAE.

What it solves

It provides a framework and dataset for creating world models that can predict future states of the environment based on previous observations and actions, which is critical for training intelligent agents like self-driving cars.

How it works

The project uses a VQ-VAE to compress video frames into discrete tokens. A world model, implemented as a Generative Pre-trained Transformer (GPT), is then trained on these tokens—using a dataset of 3 million minutes of driving video—to predict the next token in the sequence, effectively "imagining" future frames.

Who it’s for

Researchers and developers working on world models, video compression, and autonomous driving agents.

Highlights

  • Massive Dataset: Includes 100,000 minutes of compressed driving videos and a world model trained on 3 million minutes.
  • Tokenization: Uses VQ-VAE to reduce each frame to 128 tokens of 10 bits each.
  • Predictive Modeling: Employs a GPT-based architecture to predict future world states.
  • Compression Challenge: Includes a lossless compression challenge for driving video tokens.

Related

  • Project
    nvidia-cosmos/cosmos-predict1

    A suite of world foundation models for future state prediction that generates visual simulations from text or video prompts to support Physical AI development.

  • Project
    open-gigaai/giga-world-1

    An open-source framework for training and deploying robot world models to evaluate robot policies through controllable video generation.

  • Project
    simchowitzlabpublic/nano-world-model

    A minimalist framework for training video world models based on diffusion-forcing, used for future video prediction, 3D reconstruction, and robotic planning.

  • Dispatch
    VQ-Diffusion: Conditional Latent Diffusion in Discrete Space

    VQ-Diffusion is a conditional latent diffusion model that operates on a quantized discrete latent space, offering faster inference and higher image quality than traditional autoregressive models.

  • Project
    open-gigaai/giga-world-policy

    A World Action Model for robot control that uses visual dynamics during training but action-only decoding during inference to achieve real-time, low-latency deployment.