LeRobot L2D: World's Largest Open-Source Self-Driving Dataset

Hugging Face and Yaak have introduced Learning to Drive (L2D), the world’s largest open-source multimodal dataset for self-driving. L2D provides over 5,000 hours of driving data (90+ TB) across 30 German cities, specifically designed to train end-to-end AI models that can follow natural language instructions or future waypoints.

Comprehensive Multimodal Data and Scale

L2D significantly exceeds existing open-source self-driving datasets in scale and modality. While previous datasets like WAYMO, NuScenes, and COMMA focused on perception or specific planning tasks, L2D is built for end-to-end policy learning, predicting actions directly from sensor inputs.

Dataset Specifications

  • Volume: 90+ TeraBytes of data comprising 5,000+ hours of driving.
  • Visuals: 6x surrounding HD RGB cameras providing 360° coverage.
  • Vehicle State: Complete state data including speed, heading, GPS, and IMU.
  • Control Actions: Continuous data for gas, brake, and steering; discrete data for gear and turn signals.
  • Environment Context: Metadata on lane count, road type (highway, residential), road surface (asphalt, cobbled, sett), and maximum speed limits.
  • Environmental Conditions: Data on precipitation, weather (snow, clear, rain), and lighting (dawn, day, dusk).

Expert vs. Student Driving Policies

L2D distinguishes between two types of driving behaviors to provide a full spectrum of operational data:

  • Expert Policies: Executed by driving instructors with 10,000+ hours of experience. These are considered optimal with zero driving mistakes.
  • Student Policies: Executed by learner drivers with 10–50 hours of experience. These include known sub-optimalities, such as jerky steering, to provide models with examples of incorrect behavior and the reasoning behind it.

Both groups cover all EU-mandated driving tasks required for a German driving license, including navigating roundabouts, overtaking, and crossing train tracks.

Technical Implementation and Curation

Data Collection Hardware

Data was collected using 60 KIA E-niro electric vehicles equipped with an identical sensor suite, including:

  • Compute: NVIDIA Jetson AGX Xavier and Jetson Orin NX (64 GB).
  • Sensors: 6 RGB cameras, a Taoglas GNSS module, and 5G connectivity for data transmission.
  • Synchronization: All modalities are synchronized with the front-left camera at a sampling rate of 10 Hz.

Semantic Indexing and LLM-Powered Search

To manage over 1 PetaByte of raw data, Yaak implemented a semantic spatiotemporal indexing system:

  1. Map Matching: GPS traces are map-matched to the OpenStreetMap (OSM) graph using the Open-Source Routing Machine (OSRM).
  2. Route Task Assignment: Route features (e.g., bridges, tunnels), restrictions (e.g., stop signs, yield), and maneuvers (e.g., multilane left turns) are assigned to trajectories.
  3. Natural Language Search: A search system powered by llama-3.3-70b and Pydantic validation allows users to query the dataset using natural language (e.g., "drive up to the roundabout and when you have the right of way turn right") to retrieve specific episodes.

Integration with LeRobot

L2D is converted to LeRobotDataset v2.1 and v3.0 formats, enabling the use of state-of-the-art imitation learning and reinforcement learning models such as ACT, Diffusion Policy, and Pi0.

Phased Release Schedule

L2D is being released in stages to ensure episode quality:

  • R0 to R1: Initial releases focusing on basic instructions.
  • R2 to R3: Addition of route information and descriptions of suboptimal policies.
  • R4 (Planned Nov 2025): Full release of 1 million episodes and 5,000+ hours of data, including task IDs mapped to EU-mandated driving tasks.

Closed-Loop Testing and Future Evaluation

Starting in the summer of 2025, the AI community can submit models for closed-loop testing with a safety driver. Models will run in inference mode on-board the vehicle (via Jetson AGX) and will be evaluated in two modes: drive-by-waypoints (following specific coordinates) and drive-by-language (following natural language instructions).

Sources