TurboVLA – Real‑Time Vision‑Language‑Action for Robots

TL;DR – TurboVLA is a compact robot‑policy model that directly fuses visual observations and language instructions to predict continuous robot actions. It runs at 32 Hz on a consumer RTX 4090 while using < 1 GB VRAM, achieving state‑of‑the‑art success rates on the LIBERO and RoboTwin 2.0 benchmarks.

---

🎯 What the project does

• Task: Turn a robot’s camera feed + a textual command into a sequence of low‑level joint‑position actions.
• Key idea: Skip the large‑language‑model (LLM) “middle‑man”. Instead, encode vision (via DINOv3) and language (via BERT) separately, let them interact through a lightweight bidirectional module, and decode directly to action chunks.
• Result: 0.2 B parameters, 31 ms latency, 0.9 GiB VRAM, 97.6 % average success on LIBERO and 88 % on RoboTwin 2.0.

---

🏗️ Architecture at a glance

Component	Role
Vision encoder	DINOv3 ViT‑B (LIBERO) or ViT‑L (RoboTwin) – provides dense visual tokens.
Language encoder	BERT‑base uncased – produces token embeddings for the instruction.
Bidirectional vision‑language interaction	Small cross‑attention blocks that let visual and textual tokens exchange information.
Action decoder	Compact feed‑forward network that outputs continuous action chunks (12‑step for LIBERO, 50‑step for RoboTwin).

The diagram in assets/figures/architecture-overview.png shows the flow: V + L → interaction → decoder → A.

---

📊 Performance highlights

• LIBERO (4 suites, 10‑k+ episodes): 97.6 % average success, 0.2 B params, 31 ms inference.
• RoboTwin 2.0 (50 tasks, clean + randomized): 88.06 % success with a 0.4 B model, matching larger baselines.
• Speed: 32 Hz inference on an RTX 4090, using < 1 GB VRAM (bf16 precision).

Figures in assets/figures/performance.png compare latency, success, and parameter count against LLM‑centric VLA baselines.

---

📦 Getting started

1. Clone & set up environments

git clone https://github.com/H-EmbodVis/TurboVLA.git
cd TurboVLA

Create a separate Conda env for each benchmark (LIBERO or RoboTwin) – both require Python 3.10 and a CUDA‑compatible PyTorch (2.3.1).

2. Install dependencies

• LIBERO

conda create -n turbovla-libero python=3.10 -y
conda activate turbovla-libero
pip install torch==2.3.1 torchvision==0.18.1 --index-url https://download.pytorch.org/whl/cu121
pip install -e "[libero]"

• RoboTwin

conda create -n turbovla-robotwin python=3.10 -y
conda activate turbovla-robotwin
pip install -e "[robotwin]"
pip install flash-attn==2.7.4.post1 --no-build-isolation

Install the respective simulators (LIBERO, RoboTwin) separately as instructed.

3. Download model checkpoints & data

pip install -U huggingface_hub
hf download H-EmbodVis/TurboVLA --local-dir pretrained/TurboVLA

Follow the README tables to obtain DINOv3, BERT, and GroundingDINO weights, and to prepare the LIBERO TFDS/RLDS datasets or the RoboTwin converted data.

---

🏋️‍♀️ Training & evaluation

LIBERO (example)

torchrun --nproc_per_node=4 experiments/libero/train.py \
  --dataset_dir data/libero/libero_10_no_noops/1.0.0 \
  ... (other args as in README) ...

Evaluation is a single‑line call to experiments/libero/evaluate.py with the checkpoint path and suite name.

RoboTwin (example)

export ROBOTWIN_DATA_ROOT=/path/to/converted/RoboTwin
bash scripts/robotwin/train.sh   # uses env vars defined in README

Evaluation runs the policy server against the RoboTwin simulator via scripts/robotwin/evaluate.sh.

---

🤝 How to extend / contribute

• Hardware support – TODO list mentions adding Huawei Ascend NPU support.
• Model scaling – The repo already provides a 0.2 B and a 0.4 B variant; you can experiment with larger vision backbones or longer action chunks.
• New benchmarks – Plug in any vision‑language‑action dataset that follows the same observation (RGB images) + instruction format.

All code is Apache‑2.0 licensed, so you can freely adapt it for research or commercial prototypes.

---

📚 Citation

@article{xie2026turbovla,
  title  = {TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAM},
  author = {Xie, Hengyi and Yao, Chenfei and Wu, Xianjin and Xi, Xuanyang and Tang, Yiping and Xu, Di and Zhu, Yingying and Liang, Dingkang and Bai, Xiang and Ding, Han},
  journal = {arXiv preprint arXiv:2607.27205},
  year   = {2026}
}

---

Bottom line – TurboVLA shows that you don’t need a massive LLM to connect vision, language, and robot control. Its efficient design makes real‑time robot manipulation feasible on a single high‑end GPU, opening the door for on‑device or low‑cost deployment.