physical-superintelligence-lab/Psi0

[RSS26'] Welcome to Psi-Zero, a Humanoid VLA towards Universal Humanoid Intelligence.

📚 Overview

Ψ₀ (pronounced Psi-zero) is an open‑source vision‑language‑action (VLA) foundation model aimed at dexterous humanoid loco‑manipulation. It first learns visual and linguistic representations from massive egocentric video data, then is post‑trained on a smaller set of real‑world teleoperated robot demonstrations to capture the dynamics of a humanoid body (e.g., the Unitree G1). The model can be fine‑tuned on as few as ~80 new trajectories to acquire novel whole‑body skills.

🏗️ Architecture (as described in the README)

Level	Component	Role
System‑2	Vision‑language backbone	Based on Qwen3‑VL‑2B‑Instruct; extracts joint visual‑language features from camera observations and textual instructions.
System‑1	Multimodal diffusion transformer (action expert)	A ≈500 M‑parameter flow‑based diffusion transformer (inspired by Stable Diffusion 3) that predicts future whole‑body action chunks conditioned on the backbone features.
System‑0	RL‑based tracking controller	Executes the low‑level lower‑body commands produced by System‑1, providing stable physical control on the robot.

🚀 What the repo provides

Model checkpoints (Vision‑language backbone + diffusion action expert) hosted on Hugging Face.
Datasets – real‑world teleoperation recordings (9 tasks) and simulated data (SIMPLE) also on Hugging Face.
End‑to‑end training scripts for:
- Pre‑training the VLM on large egocentric datasets (EgoDex, Humanoid‑Everyday).
- Post‑training the action expert on robot data.
- Fine‑tuning on a specific robot (Unitree G1) or in the SIMPLE simulator.
Deployment utilities for serving the policy in RTC (real‑time communication) mode and for running a client on the robot.
Integration with SONIC – a whole‑body controller (forked from NV‑Labs GR00T) for more advanced robot‑side execution.
Baseline implementations (GR00T, OpenPI π₀.₅, InternVLA‑M1, H‑RDT, EgoVLA, Diffusion Policy, ACT) for comparison.
Extensive documentation: installation, data conversion (to LeRobot format), fine‑tuning, evaluation (open‑loop and in‑sim), and troubleshooting.

📦 Quick start (real‑world robot)

# Clone and set up the environment (uses uv for dependency management)
git clone git@github.com:physical-superintelligence-lab/Psi0.git && cd Psi0
curl -LsSf https://astral.sh/uv/install.sh | sh   # install uv if needed
uv venv .venv-psi --python 3.10
source .venv-psi/bin/activate
GIT_LFS_SKIP_SMUDGE=1 uv sync --group serve --group viz --group psi --active
uv pip install flash_attn==2.7.4.post1 --no-build-isolation

# Verify installation
python -c "import psi; print(psi.__version__)"

Fine‑tune on a real task (e.g., Hug box and move)

Download the pre‑collected task data from Hugging Face:

export task=Hug_box_and_move
hf download USC-PSI-Lab/psi-data g1_real_raw/$task.zip \
    --local-dir=$PSI_HOME/data/real_teleop_g1 --repo-type=dataset
unzip $PSI_HOME/data/real_teleop_g1/g1_real_raw/$task.zip -d $PSI_HOME/data/real_teleop_g1/g1_real_raw/$task

Convert to LeRobot format (script handles metadata):

python scripts/data/raw_to_lerobot.py \
    --data-root=$PWD/data/real_teleop_g1/g1_real_raw \
    --work-dir=$PWD/data/real \
    --repo-id=psi0-real-g1 \
    --robot-type=g1 \
    --task=$task

Fine‑tune (adjust CUDA_VISIBLE_DEVICES as needed):
```
scripts/train/psi0/finetune-real-psi0.sh $task
```

Serve the policy (RTC mode):

bash ./scripts/deploy/serve_psi0-rtc.sh

Run the client on the robot:
```
bash ./real/scripts/deploy_psi0-rtc.sh
```

🎮 Simulation workflow (SIMPLE)

Install SIMPLE (stand‑alone or as a submodule).
Download simulated task data (e.g., G1WholebodyXMovePickTeleop‑v0).
Fine‑tune with scripts/train/psi0/finetune-simple-psi0.sh $task.
Serve the checkpoint and run the provided evaluation notebooks or CLI scripts to obtain rollout videos.

📚 Data & Model access

Model checkpoint: https://huggingface.co/USC-PSI-Lab/psi-model
Dataset (real & simulated): https://huggingface.co/datasets/USC-PSI-Lab/psi-data
Pre‑trained VLM backbone (Qwen3‑VL‑2B‑Instruct) is automatically downloaded by the scripts/predownload_qwen3vl.py helper.

🛠️ Key dependencies

uv – fast Python package manager used throughout the repo.
flash_attn – optimized attention kernels for large transformer models.
LeRobot – common data format for robot learning pipelines.
SIMPLE – MuJoCo + Isaac Sim based humanoid benchmark.
SONIC (GR00T‑WholeBodyControl) – optional whole‑body controller for real‑robot deployment.

📄 License

The code is released under the Apache 2.0 license (see LICENSE). Model weights are also distributed under Apache 2.0, matching the repository’s licensing.

📖 Citation

If you use Ψ₀ in research, cite the accompanying arXiv paper (arXiv:2603.12263) and the repository.

TL;DR

Ψ₀ is a publicly available, large‑scale vision‑language‑action model for humanoid robots. It combines a Qwen‑based VLM with a diffusion‑style action expert, provides scripts for pre‑training, post‑training, fine‑tuning, and real‑time deployment on the Unitree G1, and includes a full simulation benchmark (SIMPLE) plus several baseline methods for comparison.