sii-research/tau-0-vla

This repo is the official implementation of "τ0-VLA: a Hierarchical Robot Foundation Model with World-Model-Guided Test-Time Computation".

What it solves

τ₀-VLA addresses the challenge of long-horizon robot manipulation tasks. It provides a hierarchical framework that can plan high-level subtasks and execute them through a generalist low-level policy that works across different robot embodiments.

How it works

The system uses a two-tier architecture:

  1. High-Level Policy: A memory-augmented planner (Proposal) that predicts the next subtask. It uses a world model to generate goal images, allowing the system to search for better alternatives when complex reasoning is required during test-time computation.
  2. Low-Level Policy: A generalist executor that converts subtasks into robot actions. It combines a Qwen3.5 vision-language backbone with a Mixture-of-Transformers action expert trained via conditional flow matching on over 40,000 hours of diverse real-world robot data.

Who it’s for

Researchers and developers working on robot foundation models, embodied intelligence, and long-horizon manipulation tasks across various robot hardware platforms.

Highlights

  • Hierarchical Structure: Separates high-level planning from low-level execution for better long-term task management.
  • World-Model Guidance: Uses a world model to generate goal images for test-time reasoning and subtask refinement.
  • Cross-Embodiment: A unified 40-dimensional state/action space allows the low-level policy to be used across different robots.
  • Extensive Training: Pretrained on a massive dataset of 40,115 hours of heterogeneous real-world robot data.

Related

  • Project
    starVLA/starVLA

    StarVLA is a modular, open‑source platform for Vision‑Language‑Action research, offering plug‑and‑play backbones, multiple action heads, and ready‑made pipelines for many robot benchmarks.

  • Dispatch
    pi0 and pi0-FAST: Vision-Language-Action Models for General Robot Control

    Hugging Face has integrated pi0 and pi0-FAST, generalist Vision-Language-Action (VLA) models developed by Physical Intelligence, into the LeRobot repository to enable versatile robot control across diverse embodiments.

  • Project
    Robbyant/lingbot-vla-v2

    LingBot‑VLA 2.0 is an open‑source Vision‑Language‑Action foundation model for robots. It unifies many robot embodiments into a 55‑dim action vector, uses MoE layers and dual‑query distillation (depth + future video) to learn predictive dynamics, and is trained on ~60 k h of robot and egocentric video data. The repo provides conda‑based installation, scripts to download 6 B‑parameter checkpoints, tools for data preparation, post‑training on custom datasets, and evaluation on the RoboTwin benchmark and real robots. Reported results show noticeable gains over the 1.0 version on bimanual and mobile manipulation tasks.

  • Project
    OpenBMB/DeepThinkVLA

    DeepThinkVLA is a Vision‑Language‑Action model that inserts an explicit chain‑of‑thought reasoning step before generating robot actions. Using a hybrid decoder (autoregressive reasoning → bidirectional parallel action generation), the authors train on a newly curated embodied CoT dataset and then refine with outcome‑driven reinforcement learning. The model achieves 97 % average success on the LIBERO benchmark, runs at only 0.175× the latency of a pure autoregressive baseline, and transfers zero‑shot to the newer LIBERO Plus suite.

  • Project
    MINT-SJTU/Evo-1

    Evo-1 is a lightweight Vision-Language-Action (VLA) model for robotic manipulation that translates visual and textual instructions into robot actions across various simulations and real hardware.