mimic-video/mimic-video

mimic‑video: Video‑Action Models for Generalizable Robot Control

What it is – A research codebase that turns large pretrained video models into Video‑Action Models (VAMs). By freezing the video backbone and training a tiny action decoder on its latent features, the system learns language‑conditioned robot policies that can be run efficiently (one video forward‑pass per action chunk). The authors demonstrate the approach on the 2 B‑parameter Cosmos‑Predict2 video model and release checkpoints for the Bridge and LIBERO robot manipulation suites.

Key components

• data_preprocessing – scripts to convert raw robot demos (Bridge, LIBERO) into safetensors, pre‑compute T5 language embeddings and optional video embeddings.
• model – dataloaders, the action‑decoder architecture, training loops and inference utilities. Built on the NVIDIA Cosmos‑Predict2 codebase and uses torchrun for multi‑node/GPU scaling.
• eval – evaluation harnesses for the SIMPLER‑Bridge environment (including a human‑in‑the‑loop variant) and the standard LIBERO benchmark.

How to get started

1. Install the UV package manager and create an environment (choose the CUDA extra that matches your GPU, e.g. uv sync --extra cu126).
2. (Optional) Pull the released checkpoints from Hugging Face after logging in with hf auth login.
3. Follow the Training section to either fine‑tune the video model or pre‑train an action decoder on your own dataset. The process involves:
   • Preparing video files and matching text instructions.
   • Running precompute_t5_embeddings.py (and optionally precompute_video_embeddings.py).
   • Editing the provided YAML config files to point at your data.
   • Launching training with torchrun -m scripts.train ….
4. Run the supplied evaluation scripts (eval.sh or eval_hil.sh) after installing the environment‑specific dependencies (Vulkan, ManiSkill2, etc.).

Typical use‑case – Researchers who have a large video foundation model and want to quickly obtain a language‑conditioned robot controller without retraining the whole video network. The framework lets you plug in new robot datasets, reuse the same video backbone, and evaluate on standard manipulation benchmarks.

License – Apache 2.0.

Citation – If you use the code or checkpoints, cite the arXiv paper:

@misc{pai2025mimicvideo,
  title={mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs},
  author={Jonas Pai and Liam Achenbach and Victoriano Montesinos and Benedek Forrai and Oier Mees and Elvis Nava},
  year={2025},
  eprint={2512.15692},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2512.15692}
}