LeRobot v0.4.0 release notes / what's new

LeRobot v0.4.0 is a major update to the open-source robotics library designed to increase the scalability, flexibility, and accessibility of robot learning. The release introduces a revamped dataset infrastructure, new high-capacity Vision-Language-Action (VLA) models, and a modular plugin system for hardware integration.

Scalable Data Infrastructure with Datasets v3.0

LeRobot v0.4.0 introduces LeRobotDataset v3.0, which overhauls the dataset infrastructure to support massive, large-scale robot learning. This version is specifically designed to handle datasets exceeding 400GB, such as Open X Embodiment (OXE) and Droid.

Key Enhancements in v3.0

  • Chunked Episode Format: Enables the handling of massive datasets at the OXE-level.
  • Streaming Capabilities: Provides faster loading times and seamless streaming of video data.
  • Unified Parquet Metadata: Replaces scattered JSON files with structured Parquet files for streamlined metadata management.
  • Performance Gains: Reduces dataset initialization times and improves memory efficiency.

Dataset Editing Tools

A new lerobot-edit-dataset CLI allows users to curate and optimize datasets through the following operations:

  • Deleting specific episodes.
  • Splitting datasets by fractions or episode indices.
  • Adding or removing features.
  • Merging multiple datasets into a single unified set.

Expanded Simulation Environments

LeRobot now provides broader support for simulation benchmarks to facilitate the training and evaluation of robotic policies.

  • LIBERO Support: Integration of LIBERO, an open benchmark for VLA policies featuring over 130 tasks, establishing a unified setup for VLA evaluation.
  • Meta-World Integration: Integration of Meta-World, a benchmark for multi-task manipulation with over 50 tasks. This is supported by a standardized use of gymnasium ≥ 1.0.0 and mujoco ≥ 3.0.0 to ensure deterministic seeding.

Codebase and Training Optimizations

Modular Data Processing Pipelines

To bridge the gap between raw sensor data and model requirements, LeRobot v0.4.0 introduces Processors. This modular pipeline system uses ProcessorStep components to handle tasks such as normalization, tokenization, and device movement.

Two specialized pipeline types are provided:

  • PolicyProcessorPipeline: Optimized for batched tensors used in high-performance training and inference.
  • RobotProcessorPipeline: Designed for real-time robot control using individual data points.

Multi-GPU Training

Training is now integrated with Hugging Face Accelerate, allowing users to scale experiments across multiple GPUs with a single command. This reduces training time proportionally to the number of GPUs used (e.g., cutting time in half with 2 GPUs).

Open-World Generalization Policies

LeRobot v0.4.0 integrates several state-of-the-art foundation models to improve robot reasoning and generalization across diverse environments.

PI0 and PI0.5

Developed by Physical Intelligence, these VLA models are designed for open-world generalization. PI0.5 is specifically noted for its ability to adapt to new environments by co-training on a diverse mix of multimodal web data, verbal instructions, and multi-environment robot data.

GR00T N1.5

Integrated through a collaboration with NVIDIA, GR00T N1.5 is a cross-embodiment foundation model. It utilizes multimodal inputs (language and images) to perform complex manipulation tasks, having been trained on a combination of real humanoid data, synthetic data from NVIDIA Isaac GR00T Blueprint, and internet-scale video data.

Hardware Integration and the Plugin System

LeRobot introduces a plugin system that allows third-party hardware to be integrated via pip install without modifying the core library. This system increases extensibility and prevents core library bloat.

New Hardware Integrations

  • Reachy 2: Support for Reachy 2 from Pollen Robotics for both real-world control and simulation.
  • Mobile Teleoperation: Using the new pipeline system, users can now teleoperate follower arms via iOS or Android devices, with the RobotProcessor handling the necessary action space transformations.

Educational Resources

Hugging Face has launched an open-source Robot Learning Course and a Modern Robot Learning Tutorial. The course covers classical robotics, generative models for imitation learning (VAEs, diffusion), and Reinforcement Learning. The tutorial provides a hands-on, first-principles approach to modern techniques with ready-to-use code examples.

Sources