thu-ml/Motus

Motus – A Unified Latent‑Action World Model

What it is – Motus is an open‑source research codebase that builds a latent‑action world model for embodied robotics. It stitches together three expert modules – a video‑generation model (VGM), a vision‑language model (VLM), and two smaller “understanding” and “action” experts – using a Mixture‑of‑Transformers (MoT) architecture. The system can be run in several modes: classic world‑model prediction, vision‑language‑action reasoning, inverse‑dynamics, video generation, or joint video‑action prediction.

Key components

Component	Base model (pre‑trained)	Size
VGM (video generation)	Wan2.2‑5B	~5 B params
VLM (vision‑language)	Qwen3‑VL‑2B	~2.1 B params
Action expert	–	~0.64 B
Understanding expert	–	~0.25 B
Total	–	~8 B

The VGM and VLM are taken from existing Hugging‑Face checkpoints; the two smaller experts are trained from scratch. Motus learns latent actions from optical‑flow “delta‑pixels”, enabling it to predict fine‑grained robot motions directly from video frames.

Training pipeline – Motus follows a three‑stage process:

1. Foundation pre‑training of the VGM and VLM on large web‑scale video and image‑text data.
2. Stage 1 – train only the VGM on egocentric human videos, synthetic data and multi‑robot trajectories.
3. Stage 2 – train the full MoT (all three experts) on the same data plus task‑agnostic data, learning the latent‑action representation.
4. Stage 3 – fine‑tune the whole system on target‑robot task trajectories (e.g., RoboTwin 2.0). The pipeline is illustrated as a six‑layer “data pyramid”.

Supported data – The repo can ingest three dataset formats:

• RoboTwin 2.0 – a simulation suite with clean and randomized splits.
• Real‑world – AC‑One and Aloha‑Agilex‑2 robot logs.
• Multi‑camera – utilities to concatenate three‑view images.

Conversion scripts (data/robotwin2/robotwin_data_convert/ and data/utils/multi_camera_concat.py) help reshape raw logs into the required directory layout (see DATA_FORMAT.md).

Running the model

• Inference – A lightweight inference script can evaluate on RoboTwin 2.0 or run a minimal demo on real‑world data. The README points to INFERENCE.md for command‑line flags and config files (configs/robotwin.yaml, configs/ac_one.yaml).
• Hardware – Inference with a pre‑encoded T5 encoder needs >24 GB VRAM (RTX 5090). Full inference without pre‑encoding or training requires 40‑80 GB GPUs (A100/H100/B200).

Checkpoints – Pre‑trained checkpoints are hosted on Hugging Face:

• Motus_Wan2_2_5B_pretrain – VGM backbone only.
• Motus – full model after latent‑action pre‑training (Stage 2).
• Motus_robotwin2 – fine‑tuned on RoboTwin 2.0 tasks (Stage 3).

The README includes a short script for downloading them with huggingface-cli and shows how to point the config files to the local paths.

Performance – On the RoboTwin 2.0 benchmark (50 clean + 500 randomized episodes per task, 50 tasks merged), Motus achieves an 87.02 % average success rate, which the authors claim is +15 % over X‑VLA and +45 % over π₀.₅.

How to get started

1. Clone the repo and create the conda environment (Python 3.10).
2. Install PyTorch 2.7.1 (CUDA 12.8) and flash‑attn for fast attention.
3. Pull the desired checkpoints from Hugging Face.
4. Follow INFERENCE.md for a quick test on RoboTwin or TRAINING.md for the full three‑stage training.

Who might use it – Researchers exploring unified models for robot perception‑action loops, developers needing a baseline that can both generate video predictions and output robot joint commands, and anyone interested in experimenting with latent‑action representations built on top of large vision‑language foundations.

Citation – The authors provide a ready‑to‑copy BibTeX entry for the arXiv pre‑print (arXiv:2512.13030).