google-research/language-table

Suite of human-collected datasets and a multi-task continuous control benchmark for open vocabulary visuolinguomotor learning.

What it solves

It addresses the challenge of creating robots that can understand and execute a wide variety of natural language instructions in real-time. Specifically, it aims to enable "open vocabulary" visuolinguomotor learning, allowing robots to perform complex, long-horizon rearrangement tasks (like making a smiley face out of blocks) based on human speech.

How it works

The project provides a framework that combines a large-scale dataset of language-annotated trajectories with a multi-task continuous control benchmark. Policies are trained using behavioral cloning on these datasets—which include hundreds of thousands of episodes from both real robots and simulations—to map visual inputs and language commands directly to motor skills.

Who it’s for

It is designed for researchers and developers working on robotics, embodied AI, and the intersection of computer vision and natural language processing.

Highlights

  • Massive Dataset: Includes nearly 600,000 language-labeled trajectories across real-world and simulated environments.
  • High Command Variety: Supports an order of magnitude more commands than previous works, with a reported 93.5% success rate on 87,000 unique natural language strings.
  • Real-Time Interaction: Enables robots to be guided by humans via real-time language for precise goals.
  • Comprehensive Assets: Provides the datasets, training scripts, environments, and pre-trained checkpoints.

Related

  • Project
    amazon-far/abc

    A scalable behavior cloning framework for robotics that uses ABC-DiT to learn physical tasks from large-scale real and simulated datasets.

  • Project
    InternRobotics/VLAC

    A Vision-Language-Action-Critic model that provides dense reward feedback and trajectory quality screening for real-world robotic reinforcement learning.

  • Project
    hzxie/DynamicVLA

    DynamicVLA is a Vision-Language-Action model and dataset designed to enable robots to manipulate objects that are moving or changing state.

  • Project
    NVIDIA/Isaac-GR00T

    An open vision-language-action (VLA) model for generalized humanoid robot skills, enabling cross-embodiment manipulation tasks via multimodal input.

  • Dispatch
    LeRobot Community Datasets: Building the ImageNet of Robotics

    Hugging Face is fostering a community-driven effort to create a diverse, open-source repository of robotics datasets via LeRobot to solve the generalization challenge in robotic policies.