Robbyant/lingbot-vla-v2

LingBot‑VLA 2.0 – Vision‑Language‑Action model for robots

What it is – An open‑source foundation model that takes visual input (images/video), language instructions, and predicts a unified 55‑dimensional action vector for many robot bodies (arms, hands, waist, head, mobile base, etc.). It is trained on ~60 k h of data that mixes robot trajectories (≈50 k h from 20 robot configurations) with egocentric human videos (≈10 k h). The model is built to bridge large‑scale pre‑training and real‑world robot deployment.

Key ideas

• Unified action representation – all embodiments are projected into a single 55‑dim vector, making the same model usable on very different robots.
• Mixture‑of‑Experts (MoE) action expert – sparse MoE layers let the network learn both universal priors and embodiment‑specific patterns while keeping compute budget fixed.
• Dual‑query distillation – current‑frame and future‑frame visual queries are distilled from a depth teacher (LingBot‑Depth) and a video‑semantic teacher (DINO‑Video), giving the model geometry and predictive dynamics.
• Predictive dynamics task – the model is also trained to forecast future visual features, improving its ability to plan ahead.
• Expanded action space – supports arms, end‑effectors, grippers, dexterous hands, waist, head, and mobile‑base signals (previous version only handled dual‑arm manipulation).

Installation (conda + PyTorch 2.8)

git clone https://github.com/Robbyant/lingbot-vla-v2.git
cd lingbot-vla-v2
bash tools/create_train_env.sh   # creates a conda env, installs flash‑attn 2.8.3

Optional flags let you point to a pre‑built flash‑attn wheel, rename the env, or force a rebuild.

Pre‑trained models – released as native‑depth checkpoints (6 B parameters). Download via the provided script or from Hugging Face / ModelScope:

python3 scripts/download_hf_model.py \
  --repo_id robbyant/lingbot-vla-v2-6b \
  --local_dir lingbot-vla

The checkpoint expects additional teacher weights (Qwen3‑VL‑4B‑Instruct, MoGe‑2‑vitb‑normal, LingBot‑Depth, DINO‑VIDEO) which are listed in the README.

Training / post‑training

1. Data preparation – create a LeRobot‑style dataset, write a robot‑config YAML that maps raw states to the 55‑dim vector, and compute normalization statistics (assets/norm_stats/*.json).
2. Run – a sample command trains on the RoboTwin 2.0 50‑task suite:

bash train.sh tasks/vla/train_lingbotvla.py ./configs/vla/robotwin/robotwin.yaml \
  --data.train_path assets/training_data/robotwin.txt \
  --data.data_name multi \
  --train.output_dir output/

Supports the Muon optimizer and its distributed variant (Distributed Muon) built on TorchTitan’s FlexShard/FSDP2.

Evaluation

• Open‑loop (offline simulation) via scripts/open_loop_eval.py – requires a robot config name (--robo_name robotwin).
• RoboTwin benchmark – a bash launcher runs all 50 tasks on a cluster of GPUs, reporting clean vs. randomized success rates.
• Real‑robot deployment – deploy.lingbot_vla_v2_policy runs a single inference call (~130 ms on an RTX 4090D with 10 denoising steps) and streams actions to the robot.

Performance highlights (as reported in the paper and README)

• GM‑100 bimanual manipulation – success ↑ from 58.2 % (VLA‑1.0) to 66.2 % (VLA 2.0) on AgileX Cobot Magic.
• Long‑horizon mobile tasks – e.g., refrigerator sorting in‑domain success 77.1 % (VLA 2.0) vs. 65.3 % (π₀.₅ baseline).
• RoboTwin 2.0 simulation – clean task success 93.5 % (VLA 2.0) vs. 88.6 % (VLA‑1.0); randomized 92.8 % vs. 86.7 %.
• Training speed – Distributed Muon cuts per‑iteration time from 5.93 s to 4.31 s on a single H20 GPU (≈ 37 % faster).

License – Apache‑2.0.

Citation – Use the provided BibTeX entry (arXiv:2607.06403, 2026).

Who might use this – Researchers building generalist robot policies, engineers needing a single model that can control many robot morphologies, and anyone interested in vision‑language‑action foundations that can be fine‑tuned on custom robot datasets.