Qwen-AgentWorld: A Language World Model for RL Environment Simulation

Overview

Qwen-AgentWorld is a world model designed to simulate reinforcement learning (RL) environments by predicting the outcomes of agent actions. Unlike traditional agents that are trained primarily to decide which action to take (policy), Qwen-AgentWorld is trained to predict what happens after an action is taken, effectively simulating the environment itself.

This approach allows for the generation of synthetic RL trajectories and the creation of adversarial training conditions without the need for expensive or slow physical sandboxes, such as Android emulators or live servers.

Core Capabilities and Supported Domains

Qwen-AgentWorld predicts the next state of an environment—such as terminal output, HTML for a webpage, or JSON for an API—based on the current state and a provided action. It operates across seven distinct domains:

• Terminal: CLI tasks and Bash commands.
• Software Engineering: Coding and development environments.
• Web Search: Interaction with search engines.
• Tools: Interaction with MCP (Model Context Protocol) tools.
• Web Browsers: General web navigation and interaction.
• Desktop OS: Operating systems including Ubuntu and Windows.
• Android OS: Mobile operating system simulation.

While other world models (like NVIDIA's Cosmos or Genie) focus on predicting visual frames or video, Qwen-AgentWorld predicts autoregressive text, making it highly efficient for technical and programmatic environments.

Impact on Agent Performance

Training agents using a language world model provides two primary advantages: simulation and improved reasoning.

High-Fidelity Simulation and Adversarial Training

Using a world model as a simulator eliminates the overhead of spinning up real sandboxes. Because the environment is simulated, developers can deliberately inject errors, hide answers, or paginate results to create adversarial conditions. This forces agents to become more robust by facing edge cases that are rarely encountered in standard, "happy path" RL environments.

Enhanced Reasoning and Self-Reflection

Teaching a model to predict the world's response encourages the habit of imagining outcomes before acting. This improves the model's reasoning and self-reflection capabilities. According to the source data, incorporating language world model RL training increased accuracy from 69.9% to 78.3% in specific tests.

The Training Pipeline

The development of Qwen-AgentWorld follows a three-stage process: "CPT injects, SFT activates, RL sharpens."

1. Continual Pre-Training (CPT)

This stage injects world knowledge. The model is fed millions of real-world action-observation trajectories from sandboxes (e.g., Android emulators, OS emulators) and world knowledge corpora covering specialized fields like law, medicine, finance, and cybersecurity.

2. Supervised Fine-Tuning (SFT)

This stage activates reasoning. The model moves beyond next-token prediction to generate explicit reasoning chains before predicting the next state. To ensure high quality, rejection sampling was used to select approximately 7,000 high-quality thinking trajectories.

3. Reinforcement Learning (RL)

This stage sharpens the fidelity of the predictions. The model is refined using on-policy rollouts and a dual-verification system to prevent reward hacking:

• LLM-as-a-Judge: Scores predictions on format, factuality, consistency, realism, and quality.
• Rule-based Verifiers: Checks for exact requirements, such as valid JSON formatting or executable code.

Practical Applications for Developers

Qwen-AgentWorld enables the creation of high-quality synthetic RL data, which can be used to fine-tune local AI models for specific use cases.

• Synthetic Trajectory Generation: Developers can use the model to generate thousands of trajectories quickly, which can then be used to distill knowledge from larger proprietary models (like Claude) into smaller, specialized local models.
• Real-time RL Environments: The model can serve as a live RL environment paired with a custom reward model for real-time agent training.
• Specialized Fine-Tuning: By adjusting system prompts (e.g., directing the model to be a pandas specialist), developers can leverage the model's internal world knowledge to generate highly accurate training data for niche technical tasks.