cheng-haha/GPT-Policy

What it solves

GPT-Policy addresses the challenge of enabling robots to adapt to unfamiliar environments and tasks without requiring gradient updates or task-specific parameter changes. It allows robots to learn from context at deployment time, overcoming the limitation where a finite set of demonstrations cannot cover every possible real-world scenario.

How it works

The system is a closed-loop control framework that connects a fixed vision-language model (VLM) to robot tools. It operates through three main components:

1. Context Compiler: Preserves task-relevant visual transitions and gathers input from human videos, robot demonstrations, target images, self-history, and human-robot interactions.
2. VLM Agent: Proposes structured robot-tool actions based on the compiled context and live observations.
3. Constrained Controller: Validates and executes the proposed actions (such as Cartesian targets or waypoint sequences), performs inverse kinematics (IK) checks, and reports the outcome back to the VLM for the next decision cycle.

Who it’s for

This project is designed for researchers and developers in embodied AI and robotics who want to implement in-context learning for robot manipulation tasks using large vision-language models.

Highlights

• In-Context Learning: Adapts to new tasks using demonstrations and feedback without retraining the model.
• Multi-Source Context: Supports human videos, robot videos with action labels, target images, and interaction history.
• Hardware Support: Includes adapters for ARX X5 and I2RT/YAM robot arm platforms.
• Closed-Loop Execution: Integrates live camera feedback and execution results to iteratively refine robot behavior.