lucidrains/pi-zero-pytorch

Implementation of π₀, the robotic foundation model architecture proposed by Physical Intelligence

What it solves

This project provides a PyTorch implementation of the $\pi_0$ (pi-zero) robotic foundation model architecture. It aims to create a general-purpose robotic policy that can generate actions based on vision, language commands, and the current state of the robot's joints.

How it works

The model is built upon a pretrained vision-language model (PaliGemma 2B) and integrates several architectural choices to generate robotic actions:

  • Flow Matching: It uses flow matching instead of traditional diffusion for policy generation, which is often more efficient.
  • Joint Attention: It employs a separation of parameters via Joint Attention (from mmDIT) to handle multimodal inputs.
  • Hybrid Attention: It utilizes Flex Attention to mix autoregressive and bidirectional attention patterns.
  • Online Learning: The EFPO class allows the model to gather experience from an environment and learn from those memories in an online fashion.

Who it’s for

  • Robotics researchers and PhD students looking to implement or experiment with the latest state-of-the-art robotic foundation models.
  • Developers building vision-language-action (VLA) models for robotic manipulation.

Highlights

  • Implementation of the $\pi_0$ architecture proposed by Physical Intelligence.
  • Support for flow matching for action generation.
  • Integration with PaliGemma 2B as a base vision-language model.
  • Built-in support for online learning via the EFPO class.

Related

  • Dispatch
    pi0 and pi0-FAST: Vision-Language-Action Models for General Robot Control

    Hugging Face has integrated pi0 and pi0-FAST, generalist Vision-Language-Action (VLA) models developed by Physical Intelligence, into the LeRobot repository to enable versatile robot control across diverse embodiments.

  • Project
    huggingface/lerobot

    A PyTorch-based library for real-world robotics that provides standardized interfaces for hardware control, scalable dataset formats, and state-of-the-art AI policies.

  • Project
    dexmal/dexbotic

    A PyTorch-based development toolbox for Vision-Language-Action (VLA) models that streamlines pretraining, fine-tuning, and evaluation for embodied intelligence.

  • Project
    FareedKhan-dev/train-llm-from-scratch

    A complete from-scratch PyTorch implementation of an LLM pipeline, covering everything from raw text tokenization and Transformer architecture to SFT and advanced RLHF alignment techniques like PPO and GRPO.

  • Project
    allenai/molmoact2

    MolmoAct2 is a family of action reasoning models that connect vision-language backbones to flow-matching action experts for real-world robot control and manipulation.