Robbyant/lingbot-va

What it solves

LingBot-VA addresses the challenge of simultaneous world modeling and action inference in robot control. It aims to improve sample efficiency, long-horizon success rates, and the ability of robots to generalize to novel scenes by unifying visual dynamics prediction and action generation.

How it works

The project implements an autoregressive (AR) diffusion framework. It uses a dual-stream Mixture-of-Transformers (MoT) architecture that interleaves visual dynamics and action inference within a single sequence. To ensure high-efficiency execution, it employs Asynchronous Execution and a KV Cache. The system operates on video latent representations (processed via a Wan2.2 VAE) rather than raw pixels, and supports post-training on custom robotic manipulation datasets using FSDP for distributed training.

Who it’s for

This framework is designed for robotics researchers and developers working on robot control, visual world modeling, and embodied AI, specifically those looking to fine-tune models on custom manipulation datasets.

Highlights

• Unified Architecture: Combines visual dynamics prediction and action inference in one interleaved sequence.
• Efficient Execution: Utilizes a dual-stream MoT architecture with KV Cache and Asynchronous Execution.
• High Performance: Achieves state-of-the-art success rates on RoboTwin 2.0 and LIBERO benchmarks.
• Customizable: Provides a complete pipeline for post-training on custom datasets, including data conversion to LeRobot format and latent extraction.