LeRobotDataset v3.0 release notes / what's new

Hugging Face has released LeRobotDataset:v3.0, a standardized dataset format for robot learning designed to scale to millions of episodes. This update solves file-system limitations found in version 2.0 by packing multiple episodes into single files and introduces a native streaming mode for processing large datasets without full local downloads.

Scalable Data Storage Architecture

LeRobotDataset:v3.0 transitions from a one-episode-per-file approach to a concatenated storage model to reduce pressure on local and remote file systems. The format organizes data into three primary pillars:

  • Tabular Data: Low-dimensional, high-frequency data (such as joint states and actions) are stored in Apache Parquet files. These are accessed via the datasets library for fast memory-mapped or streaming access.
  • Visual Data: Camera frames are concatenated and encoded into MP4 files. Multiple episodes are grouped into a single video file, and multiple videos are grouped by camera view. To further optimize file system performance, these are organized into subdirectories.
  • Metadata: JSON files serve as the relational layer, mapping the tabular and visual data. This includes feature schemas, frame rates, normalization statistics, and episode boundaries.

Dataset Repository Structure

To maintain retrieval efficiency at the individual episode level despite concatenated storage, the repository uses the following structure:

  • meta/info.json: The central schema defining features (e.g., observation.state, action), shapes, data types, FPS, and path templates.
  • meta/stats.json: Aggregated statistics (mean, std, min, max) for each feature, used for data normalization.
  • meta/tasks.jsonl: A mapping of natural language task descriptions to integer indices for task-conditioned policy training.
  • meta/episodes/: Episode-specific metadata (length, task, data pointers) stored in chunked Parquet files.
  • data/: Core frame-by-frame tabular data in Parquet files, where data from multiple episodes are concatenated into larger files.
  • videos/: MP4 files containing concatenated footage from multiple episodes, organized by camera key and chunk.

Streaming and Data Access

LeRobotDataset:v3.0 introduces the StreamingLeRobotDataset interface, which allows users to process batches of data on the fly directly from the Hugging Face Hub without downloading the entire collection to disk.

For standard local access, the LeRobotDataset class integrates with the PyTorch DataLoader. It supports a native windowing operation via the delta_timestamps argument, allowing users to retrieve a stack of observations and actions from a specific number of seconds before and after a given frame—a critical requirement for reinforcement learning (RL) and behavioral cloning (BC) algorithms.

Migration and Installation

LeRobotDataset:v3.0 will be officially integrated into the lerobot-v0.4.0 stable release. Users can currently access the format via a pre-release version of lerobot-v0.3.x.

To migrate existing v2.1 datasets to v3.0, Hugging Face provides a conversion utility that aggregates individual episode files into the new concatenated format and updates the metadata:

python -m lerobot.datasets.v30.convert_dataset_v21_to_v30 --repo-id=<HFUSER/DATASET_ID>

Supported Embodiments

The format is designed to be extensible and currently supports a wide range of robotics data, including:

  • Manipulator platforms (e.g., SO-100 arms, ALOHA-2 setup)
  • Real-world humanoid data
  • Simulation datasets
  • Self-driving car data

Sources