LeRobot v0.6.0 release notes / what's new

LeRobot v0.6.0 focuses on closing the robot learning loop by integrating policies that can imagine future states, reward models for success detection, and a deployment CLI that transforms failures into training data. This release expands the ecosystem with six new simulation benchmarks, depth sensing support, and cloud training capabilities via Hugging Face Jobs.

World Models and Predictive Policies

LeRobot v0.6.0 introduces three policies designed to test whether world models improve robot performance by learning to imagine future states during training.

VLA-JEPA

Built on Qwen3-VL-2B, VLA-JEPA uses a JEPA world model to predict future frames in latent space based on the model's actions. The world model is used only during training and is discarded at inference, resulting in zero additional inference cost. Pretrained base checkpoints, including one for DROID, are available on the Hugging Face Hub.

LingBot-VA

LingBot-VA is an autoregressive video-action model that predicts future video and actions in chunks. It incorporates real observations to ground its imagination. Users can save the robot's imagined rollouts via the --policy.save_predicted_video=true flag to compare them against actual outcomes. Inference requires a GPU with 24–32 GB of VRAM.

FastWAM

FastWAM combines a ~5B video-generation expert and a compact action expert within a single network. While it learns to "dream" rollouts during training, it skips this process at inference to directly denoise action chunks. It can be fine-tuned from the lerobot/fastwam_base checkpoint.

Expanded Vision-Language-Action (VLA) Model Zoo

This release significantly grows the library of supported VLAs:

  • GR00T N1.7: An upgrade to NVIDIA's cross-embodiment foundation model, replacing N1.5. It utilizes Cosmos-Reason2-2B (based on Qwen3-VL) with a flow-matching action head. Integration is parity-tested against NVIDIA's original Isaac-GR00T implementation.
  • MolmoAct2: A VLA from the Allen Institute for AI. LeRobot now supports its full lifecycle, including LoRA fine-tuning and zero-shot deployment on SO-100/101 robots. Inference fits within ~12 GB of VRAM at bf16.
  • EO-1: A VLA pretrained on interleaved vision-text-action data, featuring a Qwen2.5-VL-3B backbone and a flow-matching action head.
  • Multitask DiT: A ~450M-parameter diffusion transformer conditioned on CLIP vision and language embeddings, allowing a single model to learn multiple tasks via natural language.
  • EVO1: A compact 0.77B parameter model using an InternVL3-1B backbone and a flow-matching action head, designed for real-time execution on modest GPUs.

Reward Models and Success Detection

LeRobot v0.6.0 introduces a unified reward models API (lerobot.rewards) to provide success detection and progress estimation.

Robometer

Robometer is a general-purpose reward model built on Qwen3-VL-4B, trained on over one million robot trajectories. It scores task progress and success from raw video and language instructions without requiring task-specific training.

TOPReward

TOPReward provides zero-shot reward estimation by wrapping an off-the-shelf VLM (Qwen3-VL) and reading the log-probability of the "True" token given a trajectory and instruction.

Dataset Enhancements and Data Loading

Improvements to the dataset pipeline focus on flexibility, richness, and speed:

  • Custom Video Encoding: The --dataset.rgb_encoder.* options allow users to specify codecs, quality, and pixel formats. The vcodec=auto setting automatically probes for hardware encoders (NVENC, VideoToolbox, VAAPI, QSV) before falling back to AV1.
  • End-to-End Depth Support: LeRobot now supports recording depth maps (captured in millimeters and compressed as 12-bit depth video streams) across various robots, including the SO-100/101, Koch, and Unitree G1.
  • VLM-Powered Annotations: The lerobot-annotate CLI uses VLMs (such as Qwen2.5-VL-7B-Instruct) to automatically generate timestamped subtasks, plans, and VQA pairs for datasets.
  • Performance Gains: Data loading is up to 2x faster due to parallel multi-camera frame decoding and the use of compact uint8 frames. Loading specific episode subsets has been reduced from 275 seconds to 0.06 seconds.

Unified Evaluation Benchmarks

Six new simulation benchmarks are now accessible via the lerobot-eval CLI, each providing a Docker image and a SmolVLA baseline:

  • LIBERO-plus: Tests VLA robustness via 10,000 perturbed variants of LIBERO.
  • RoboTwin 2.0: Features 50 bimanual manipulation tasks on SAPIEN with heavy domain randomization.
  • RoboCasa365: Covers 365 kitchen tasks across 2,500 procedurally generated kitchens.
  • RoboCerebra: Evaluates long-horizon behavior using chained sub-goals.
  • RoboMME: A memory exam testing counting, object tracking, and procedure imitation.
  • VLABench: Tests manipulation knowledge and reasoning.

Training and Deployment Workflows

lerobot-rollout and DAgger

Deployment is now a dedicated workflow via the lerobot-rollout CLI. A key feature is the DAgger strategy, which allows a human operator to intervene when a policy fails, record the correction using a leader arm, and tag the frame as an intervention. This data is then used for subsequent fine-tuning.

Scalable Training

  • FSDP Support: Fully Sharded Data Parallel (FSDP) via Accelerate allows training of models that exceed single-GPU memory. Parameters, gradients, and optimizer states are sharded across GPUs.
  • HF Jobs: The lerobot-train command can now be executed in the cloud using the --job.target flag, supporting hardware from T4 to 8x H200 GPUs.

Codebase and Ecosystem

  • Leaner Installation: Base dependencies are reduced by 40%, with feature-scoped extras (e.g., [training], [evaluation]) used to minimize the install footprint.
  • Visualization: Integration with Foxglove allows for streaming teleoperation and rollouts via --display_mode=foxglove.
  • LeLab: A browser-based GUI that enables calibration, recording, and training without using the CLI, currently supporting the SO-ARM101.
  • Isaac Teleop: A collaboration with NVIDIA allowing SO-101 teleoperation via VR controllers through the Isaac Teleop stack.

Sources