InternRobotics/VLAC

What it solves

VLAC is designed to improve real-world robotic reinforcement learning and data refinement. It addresses the challenge of providing dense, accurate reward feedback and filtering low-quality trajectories in robotic manipulation tasks, which is often difficult to achieve with traditional methods.

How it works

VLAC is a Vision-Language-Action-Critic model trained on a massive dataset of human egocentric data, public robotic manipulation data, and self-collected data. It uses a pair-wise comparison mechanism to evaluate task progress and recognize state changes between frames. By combining image inputs and task descriptions, the model can predict task progress, verify task completion, and distinguish between successful and failed actions.

Who it’s for

This project is for robotics researchers and developers working on embodied intelligence, specifically those focusing on reinforcement learning, imitation learning, and the need for high-quality trajectory data for training robots.

Highlights

• Pair-wise Comparison: Improves accuracy in dense critic rewards and state change recognition.
• Multi-modal Capabilities: Supports process tracking, task completion judgment, VQA, and embodied action output.
• Zero-shot/One-shot Generalization: Maintains performance across new entities, scenarios, and tasks without additional training.
• Trajectory Quality Screening: Filters out low-score trajectories and masks negative actions to improve imitation learning efficiency.
• Human-Task Synesthesia: Leverages human egocentric data (Ego4D) to better understand real-world human tasks.