LeRobotDataset video‑encoding format reduces robotics dataset size and speeds up training

TL;DR

Hugging Face introduced the LeRobotDataset format, which stores robot visual data as compressed video streams, cutting dataset size to an average of 14 % of the original while keeping loading times fast and training performance unchanged.

What is a robotics dataset and why video helps

A robotics dataset for end‑to‑end learning consists of two modalities: (1) a visual stream (camera images) and (2) proprioceptive or goal‑position vectors (state/action). Historically the visual frames were saved as individual PNG files, which is highly redundant because consecutive frames share large regions of similarity. Video codecs exploit spatial compression (like JPEG) and temporal compression (storing only inter‑frame differences) to achieve compression ratios of 1:20 or better. By encoding the visual modality as a video, the same information can be stored in a fraction of the original size while still being decoded quickly for training.

The LeRobotDataset contribution

The new LeRobotDataset format is:

  • Simple – a single video file per episode plus a lightweight metadata file.
  • Lightweight – average storage is 14 % of the raw image size (some datasets shrink to <1 %).
  • Fast to load – decoding a single frame is comparable to loading a PNG; decoding multiple successive frames costs only 25 %–50 % of the time required for the same number of PNGs.
  • Easy to share – native integration with the Hugging Face Hub.
  • Easy to visualize – interactive Spaces let users browse the video and associated actions.

How video encoding works (codec basics)

Video encoding reduces size through two mechanisms:

  1. Spatial compression – compresses each frame using patterns within the image (e.g., large sky regions).
  2. Temporal compression – stores only the differences between frames (P‑frames/B‑frames) relative to periodic keyframes (I‑frames).

The encoded bitstream is then packaged in a container such as MP4. The authors built a benchmark (see the GitHub repo) to evaluate different codec choices, pixel formats, GOP sizes, and constant‑rate‑factor (CRF) settings.

Evaluation criteria

Four criteria guided the design:

  • Size – smaller files reduce storage and download costs.
  • Decoding time – faster frame extraction speeds up training.
  • Quality – high visual fidelity is needed to avoid degrading policy performance.
  • Compatibility – videos must play on browsers and common media players; the pixel format yuv420p satisfies this requirement.

Benchmark variables and datasets

The study used four representative datasets covering a range of resolutions and scene dynamics:

Dataset Resolution Scene type
lerobot/pusht_image 96 × 96 Simulated geometric shapes
aliberts/aloha_mobile_shrimp_image 480 × 640 Real‑world indoor, moving camera
aliberts/paris_street 720 × 1280 Real‑world outdoor, moving camera
aliberts/kitchen 1080 × 1920 Real‑world indoor, fixed camera

Encoding parameters explored included three codecs (libx264, libx265, libsvtav1), two pixel formats (yuv444p, yuv420p), GOP sizes from 1 to 40, and CRF values from 0 (lossless) to 50 (highly lossy).

Selected settings for production (v1.6)

After the benchmark, the team settled on the following configuration for the released datasets (v1.6):

  • Codec: libsvtav1 (AV1 implementation)
  • Pixel format: yuv420p
  • GOP size: 2 (keyframe every other frame)
  • CRF: 30 (balanced quality/compression)

These settings improved visual quality compared with the previous libx264‑based v1.5 while preserving compatibility.

Size reductions achieved

Across 70+ public robot datasets, the average compression ratio is 14 % of the original size. Some datasets shrank to 0.2 % of their raw size when the source images were uncompressed. Example reductions:

  • lerobot/nyu_rot_dataset: 5.3 MB → 318 KB (5.8 %)
  • lerobot/utokyo_xarm_pick_and_place: 1.3 GB → 54.6 MB (4.1 %)
  • lerobot/berkeley_gnm_recon*: 18.7 GB → 29.3 MB (0.2 %)

A full table is provided in the original blog post.

Loading‑time performance

Loading time scales much better with resolution when using video. In the advantageous scenario (multiple consecutive frames), decoding time is 4 %–48 % of the PNG baseline, depending on resolution and codec. Graphs in the source show a clear advantage for 2‑frame and 6‑frame batches.

Quality metrics

The authors measured three standard image‑quality metrics on decoded frames:

  • MSE (lower is better)
  • PSNR (higher is better)
  • SSIM (higher is better)

For the four benchmark datasets, libsvtav1 with yuv420p and crf=30 consistently achieved the best PSNR and SSIM while keeping MSE low. For example, on the aliberts/kitchen dataset, AV1 attained PSNR = 39.20 dB and SSIM = 96.84 %, outperforming both H.264 and H.265.

Impact on policy training

Training curves for two representative policies—Diffusion on the pusht dataset and ACT on an aloha dataset—showed no degradation when using the video‑encoded format. The curves overlap with those obtained from raw PNG data, confirming that compression does not harm learning.

Future directions

The current benchmark omitted several encoding knobs (e.g., -preset, -tune, two‑pass encoding) and alternative decoders (torchcodec, decord, etc.). Exploring these could further improve the size‑quality‑speed trade‑off. Additionally, encoding depth maps alongside RGB frames remains an open problem.

Why this matters

By compressing visual data into efficient video streams, Hugging Face makes large‑scale robot learning datasets practical to store, share, and train on. The approach narrows the gap between robotics and other AI domains that already benefit from massive internet‑scale data, potentially accelerating research in end‑to‑end robot policy learning.

Sources