yaoyao-jpg/PhiZero

What it solves

PhiZero is a world model designed to reason about future physical dynamics. It addresses the challenge of predicting how objects move and interact in the real world by first reasoning in a discrete "Physical Language" before rendering the result as a video.

How it works

The system operates in two primary stages: reasoning and rendering. A Physical Language Reasoner (based on Qwen3-VL) predicts a sequence of 256 discrete motion tokens (Physical Language) based on a first frame and a text caption. A Diffusion Decoder then takes these tokens and the first frame to render the predicted motion into a video. The model can be trained via a multi-step workflow: training a tokenizer to create the Physical Language, extracting ground truth tokens from videos, training the reasoner to predict those tokens, and optionally adapting the decoder for specific domains like robotics.

Who it’s for

This project is aimed at researchers and developers working on world models, physical AI, and video generation, particularly those interested in motion transfer (e.g., human-to-robot or sim-to-real) and physical reasoning.

Highlights

• Physical Language: Uses a discrete tokenized representation of motion to separate reasoning from rendering.
• Motion Transfer: Supports transferring motion from a source video to a different target object or environment (e.g., ego-to-dexterous robot).
• Reason-then-Render: Decouples the physical prediction of movement from the final pixel generation.
• Domain Adaptation: Includes tools to adapt the model to specific physical domains through reconstruction training.