Qwen-VLA: Unifying Vision-Language-Action Modeling for Embodied Intelligence

Qwen-VLA is a general-purpose Vision-Language-Action (VLA) model designed to transition multimodal large language models from purely understanding the world to acting within it. By extending visual perception, language understanding, and spatial reasoning into continuous action generation and trajectory prediction, Qwen-VLA enables a single model to handle diverse robotic tasks across different embodiments and environments.

Unified Framework for Embodied Tasks

Qwen-VLA replaces specialized, task-specific models with a unified generalist policy. It formulates robotic manipulation and vision-language navigation under a single framework: the model predicts the next action or trajectory based on visual observations, language instructions, and embodiment-specific conditions. This architecture utilizes a Qwen multimodal backbone for understanding and a dedicated action decoder for generating continuous actions.

Four-Stage Training Pipeline

Qwen-VLA is developed through a joint training system that progresses from learning language priors to achieving closed-loop control across four distinct stages:

Stage I: Text-to-Action (T2A) Pretraining

In this initial stage, the VLM is frozen and only the action decoder is trained on language and embodiment prompts. This process treats the generation of high-dimensional continuous trajectories from simple language instructions (e.g., "pick up the red cup") as a form of decompression from language to action.

Stage II: Continual Pretraining (CPT)

Both the VLM and the action decoder are unfrozen and jointly trained on a full multimodal data mixture. This stage grounds the language-action priors established in T2A within concrete visual scenes, resulting in the Qwen-VLA-Base model.

Stage III: Supervised Fine-Tuning (SFT)

The model branches into two tracks from the CPT checkpoint:

  • Multi-task SFT: Joint fine-tuning on manipulation, navigation, VQA, and spatial grounding.
  • Real-robot SFT: Fine-tuning on internal teleoperation data for physical deployment.

Stage IV: Reinforcement Learning (RL)

Using PPO, the model is optimized for closed-loop task success within the SimplerEnv simulation. This produces the final Qwen-VLA-Instruct model, with gains that transfer to unseen environments and robot embodiments.

Training Data Composition

The model is trained on a massive dataset spanning five major sources:

  • Robot Manipulation: Over 10,000 hours of public data, 1,000+ hours of internal real-robot trajectories, and 8 million+ synthetic simulation trajectories.
  • Human Egocentric Data: Data from Ego4D, EPIC-KITCHENS, Xperience, EgoDex (829 hours), and EgoVerse (1,300+ hours, 1,965 tasks, 240 scenes).
  • Synthetic Simulation Data: 359,848 successful trajectories across 20 tabletop scenes, 200 configurations, and 450 tasks.
  • Text-to-Action Data: Approximately 7.2 million trajectories (14,000+ hours) across 6 templates and 6 single-arm robots.
  • Vision-Language Navigation: Data for long-horizon trajectory planning and instruction following.
  • General Vision-Language Data: Used to preserve spatial grounding and multimodal understanding, including 48,000 fine-grained action descriptions annotated across 13 dimensions.

Performance and Benchmarks

Qwen-VLA-Instruct demonstrates that a single generalist model can match or exceed the performance of specialist models across multiple benchmarks:

Benchmark Best Specialist Model Qwen-VLA
LIBERO ABot-M0 (98.6%) 97.9%
RoboCasa-GR1 ABot-M0 (58.3%) 56.7%
Simpler-WidowX StarVLA-OFT (64.6%) 73.7%
RoboTwin-Easy / Hard ABot-M0 (86.0% / 85.0%) 86.1% / 87.2%

Additionally, Qwen-VLA-Instruct achieves a 69.0% Oracle Success Rate and 57.5% Success Rate on R2R Val-Unseen, and 59.6% SR and 47.8% SPL on RxR Val-Unseen for vision-language navigation (VLN-CE), surpassing all open-source baselines.

Real-World Generalization and Dynamic Scenes

Qwen-VLA exhibits strong out-of-distribution (OOD) generalization in real-world ALOHA dual-arm experiments. The pretrained model achieves 83.6% average in-domain success and 76.9% average OOD success, significantly outperforming models trained from scratch (48.5% / 36.2%) and $\pi_{0.5}$ (71.6% / 41.5%).

The model generalizes to unseen colors, objects (such as vegetables or sunglasses), backgrounds, and lighting conditions. It also handles compositional tasks, such as "tidy up the table," by identifying multiple targets and executing multi-step operations.

In dynamic manipulation tasks using the DOMINO benchmark, Qwen-VLA-Instruct achieves a 26.6% success rate and a 39.5 manipulation score without specific fine-tuning, outperforming several standard VLA baselines and some specialist models for dynamic manipulation.

Sources