amazon-far/abc

What it solves

It provides a scalable framework for behavior cloning in robotics, allowing models to learn complex physical tasks (such as organizing bottles in a bin) from large-scale open datasets of real and simulated robot trajectories.

How it works

The project implements a training pipeline centered around ABC-DiT, a model that uses a vision backbone (DINOv3) and a text encoder (CLIP) to process visual and linguistic inputs. It utilizes flow-matching for action prediction and supports training on a mixture of simulated and real-world data. The system includes tools for converting raw MCAP data into a specialized format (binary state-action files and efficient MP4 video stacks) to optimize data loading during training.

Who it’s for

Robotics researchers and engineers who want to train vision-language-action (VLA) models using large-scale behavior cloning and evaluate them in both simulation and real-world environments.

Highlights

• Scalable Training: Supports multi-GPU training (e.g., 8x H100/H200) with optimized throughput.
• Hybrid Data Support: Capable of training on both simulated and real-world data to improve generalization.
• Efficient Data Pipeline: Includes conversion scripts to transform raw robot data into high-performance formats for faster training.
• Integrated Evaluation: Provides a simulation evaluation suite with visualization via Viser and systematic success-rate logging.