Qwen-RobotWorld: Boundless Worlds for Embodied Agents

Qwen-RobotWorld is a unified world model that treats natural language as a universal action interface to bridge the gap between general video generation and domain-specific embodied models. By standardizing 20+ robot embodiments and 500+ action categories into a single language-driven interface, the model enables joint training across manipulation, autonomous driving, and indoor navigation, allowing physical knowledge from one domain to reinforce others.

Dual-Stream Diffusion Architecture

Qwen-RobotWorld utilizes a dual-stream Multimodal Diffusion Transformer (MMDiT) to integrate semantic understanding with visual generation. The architecture consists of two primary streams:

  • Understanding Stream: Processes semantic features from a frozen Qwen2.5-VL encoder to represent the language action $a_t$.
  • Generation Stream: Processes visual latents from a video-compatible VAE to represent the visual state $s_t$.

These streams interact via joint attention at every layer, facilitating bidirectional cross-modal fusion. By using a full MLLM (Qwen2.5-VL) as the action encoder instead of lightweight encoders like CLIP or T5, the model achieves deeper language understanding for complex instructions and leverages internalized world knowledge to ensure physically plausible transitions, such as maintaining the rigidity of robot arms.

Cross-Embodiment and Multi-View Generation

Scene2Robot Human-to-Robot Transfer

The Scene2Robot mechanism enables cross-embodiment video editing by retargeting human demonstrations to 14 different robot morphologies. This is achieved through a multi-segment conditioning mechanism where joint attention allows the model to simultaneously attend to scene appearance and robot motion trajectories. This capability serves as both a training data scaling engine and an inference-time transfer tool.

Geometrically Consistent Multi-View Generation

To eliminate occlusions common in single-camera views, Qwen-RobotWorld generates 2–4 synchronized camera streams (main view, wrist-mounted views, and third-person views). The model achieves 3D-consistent object identity and motion trajectories without architectural modifications by spatially concatenating synchronized frames during training and using asymmetric 3D RoPE for spatial encoding. This multi-view consistency acts as a geometric regularizer that teaches the model object shape, depth, and spatial layout.

Embodied World Knowledge (EWK) Dataset

The model is trained on the Embodied World Knowledge (EWK) dataset, which comprises 8.6M cross-scenario training pairs organized across four axes:

  • Multi-Embodiment: Covers 20+ robot models, including human hands, 7 robot arm configurations, ego vehicles, and mobile agents.
  • Multi-Task: Includes 500+ action categories spanning atomic manipulation, long-horizon compositions, and locomotion.
  • Multi-Scenario: Combines real-world data (kitchens, workshops, outdoors) with photorealistic simulations.
  • Multi-View: Approximately 1.6M of the 6M embodied samples include 2–4 view concatenations.

Action-Language Mapping

To resolve representational heterogeneity (e.g., the difference between joint angles in manipulation and steering commands in driving), Qwen-RobotWorld projects all action signals onto a shared natural language space. A five-layer annotation pipeline ensures precision:

  1. Task Goal: High-level intent.
  2. Action Detail: Spatio-temporal trajectories and viewpoint declarations.
  3. Physical Feedback: Observable environmental consequences.
  4. Comprehensive Caption: Full description for precise prediction.
  5. Concise Caption: Essential elements for brief commands.

Training Curriculum

The model follows a general-to-expert progressive curriculum to build foundational priors before specializing in embodied physics:

Phase Data Mix Objective
Pretraining T2I / T2V / TI2V joint + Human interaction (Ego4D, EPIC-Kitchen) Build foundational visual priors and grasping/tool-use priors
SFT Phase 1 Embodied + general joint training Core manipulation physics
SFT Phase 2 Multi-view expansion Broaden viewpoint coverage
SFT Phase 3 Multi-view concatenation Cross-view geometric consistency
SFT Phase 4 Complex cross-domain Long-horizon & cross-scenario generalization

Performance and Benchmarks

Qwen-RobotWorld outperforms both general video generation models (e.g., Sora2, Veo3) and specialized embodied world models (e.g., Cosmos, LVP) across four key benchmarks:

  • EWMBench (4.60): Demonstrates strong motion fidelity (HSD 0.566) and scene consistency (0.914).
  • DreamGen (4.952): Shows strong object-level compositional generalization (GR1-Object IF: 0.878).
  • WorldModelBench (8.99): Achieves perfect physics adherence (1.00) across Newton's laws, mass conservation, fluid dynamics, and gravity.
  • PBench (0.804): Exhibits strong domain understanding (0.857) and motion smoothness (0.990).

Capabilities and Generalization

The model demonstrates robust zero-shot capabilities and cross-domain generalization:

  • Fine-Grained Grounding: Produces distinct videos based on single-keyword changes in instructions (e.g., "Pick up the red strawberry" vs. "Pick up the yellow potato").
  • Cross-Domain Mobility: Generalizes beyond manipulation to autonomous driving (using Bench2Drive, Waymo, etc.) and indoor navigation (using VLNVerse).
  • Long-Horizon Reasoning: Handles complex, multi-step instructions such as sequentially picking up multiple objects and placing them in a specific order.

Sources