NVlabs/ProtoMotions

What it solves

ProtoMotions3 is a GPU-accelerated framework designed to bridge the gap between animation and robotics. It provides a fast prototyping platform for training physically simulated digital humans and humanoid robots to perform complex motor skills, enabling researchers to move from large-scale motion learning in simulation to real-world hardware deployment.

How it works

The framework utilizes GPU acceleration to enable large-scale reinforcement learning (RL) and motion imitation. It supports multiple physics engines (NVIDIA Newton, IsaacLab, IsaacGym, MuJoCo, and Genesis) and provides tools for retargeting motion data (via PyRoki) from human datasets like AMASS to specific robot morphologies. It uses a modular design where control, observation, and reward functions are decoupled, allowing users to easily swap robots, simulators, or RL algorithms.

Who it’s for

This tool is intended for researchers and practitioners in robotics, computer animation, and reinforcement learning who are developing humanoid control policies.

Highlights

• Rapid Motion Learning: Train characters on 40+ hours of human animation data (AMASS) in under 12 hours using 4 A100 GPUs.
• Sim-to-Real Pipeline: Export policies as ONNX models for zero-shot transfer to real hardware, such as the Unitree G1 humanoid robot.
• Cross-Simulator Testing: One-click testing of policies across different physics engines to ensure robustness (Sim2Sim).
• Extensible Architecture: Modular components for procedural scene generation, custom robot integration via MuJoCo XML, and support for generative policies like MaskedMimic.
• Text-to-Motion Integration: Compatible with NVIDIA's Kimodo model to train physics-based policies from text prompts.