Robbyant/lingbot-va

[RSS 2026] Causal video-action world model for generalist robot control

What it solves

LingBot-VA addresses the challenge of simultaneous world modeling and action inference in robot control. It aims to improve sample efficiency, long-horizon success rates, and the ability of robots to generalize to novel scenes by unifying visual dynamics prediction and action generation.

How it works

The project implements an autoregressive (AR) diffusion framework. It uses a dual-stream Mixture-of-Transformers (MoT) architecture that interleaves visual dynamics and action inference within a single sequence. To ensure high-efficiency execution, it employs Asynchronous Execution and a KV Cache. The system operates on video latent representations (processed via a Wan2.2 VAE) rather than raw pixels, and supports post-training on custom robotic manipulation datasets using FSDP for distributed training.

Who it’s for

This framework is designed for robotics researchers and developers working on robot control, visual world modeling, and embodied AI, specifically those looking to fine-tune models on custom manipulation datasets.

Highlights

  • Unified Architecture: Combines visual dynamics prediction and action inference in one interleaved sequence.
  • Efficient Execution: Utilizes a dual-stream MoT architecture with KV Cache and Asynchronous Execution.
  • High Performance: Achieves state-of-the-art success rates on RoboTwin 2.0 and LIBERO benchmarks.
  • Customizable: Provides a complete pipeline for post-training on custom datasets, including data conversion to LeRobot format and latent extraction.

Related

  • Project
    EternalEvan/Astra

    Astra is an interactive world model that uses an autoregressive diffusion transformer to generate realistic, action-conditioned long-horizon video predictions.

  • Project
    Robbyant/lingbot-vla-v2

    LingBot‑VLA 2.0 is an open‑source Vision‑Language‑Action foundation model for robots. It unifies many robot embodiments into a 55‑dim action vector, uses MoE layers and dual‑query distillation (depth + future video) to learn predictive dynamics, and is trained on ~60 k h of robot and egocentric video data. The repo provides conda‑based installation, scripts to download 6 B‑parameter checkpoints, tools for data preparation, post‑training on custom datasets, and evaluation on the RoboTwin benchmark and real robots. Reported results show noticeable gains over the 1.0 version on bimanual and mobile manipulation tasks.

  • Project
    hao-ai-lab/FastVideo

    FastVideo is an open‑source framework for training, fine‑tuning, and real‑time inference of video diffusion models. It offers sparse‑attention distillation, scalable multi‑GPU training, and cross‑platform support (NVIDIA GPUs, Apple Silicon via MLX, ARM DGX Spark). The library includes a CLI, Python API, and the Dreamverse app for live video editing. Install via `uv` and start generating videos with a few lines of Python.

  • Project
    ShuangLI59/unified_video_action

    A unified model for robotic manipulation that combines video generation and action prediction through a two-stage training process to improve visual foresight and task execution.

  • Project
    Robbyant/lingbot-map

    LingBot‑Map is a feed‑forward transformer that reconstructs 3‑D point clouds from streaming video in real‑time, using a paged KV‑cache (FlashInfer) and optional sky‑masking. It provides interactive and offline demo scripts, pretrained checkpoints, and supports very long sequences via windowed inference.