mll-lab-nu/VAGEN

World model reinforcement learning for multi-turn VLM agents. RL for vision framework (NeurIPS 2025).

What it solves

VAGEN addresses the difficulty of training Vision-Language Model (VLM) agents to perform complex, multi-turn tasks. Standard reinforcement learning often rewards only the final task success, which provides sparse rewards and makes it hard for agents to learn the underlying logic of their environment. VAGEN solves this by reinforcing the agent's internal "world model" reasoning—specifically its ability to estimate the current state and predict future transitions—rather than just the end result.

How it works

VAGEN frames multi-turn agentic tasks as a Partially Observable Markov Decision Process (POMDP). It uses a novel "World Modeling RL" approach that decomposes reasoning into two explicit steps: StateEstimation ("what is the current state?") and TransitionModeling ("what comes next?").

To train this, the framework employs:

  • WorldModeling Reward: A turn-level reward provided by an LLM-as-Judge to evaluate the reasoning process.
  • Bi-Level GAE: A credit assignment mechanism for turn-aware learning.
  • Flexible Harnesses: Different ways of managing conversation history (concat, no-concat, and compact), where "compact" summarizes long conversations to fit within token budgets.
  • Modular Architecture: Decoupled environments, harnesses, and training backends (supporting vLLM and SGLang).

Who it’s for

  • AI researchers and developers building VLM agents for visual environments.
  • Developers working on robotics, navigation, and spatial reasoning tasks (e.g., Sokoban, ManiSkill).
  • ML engineers looking for a scalable RL framework for multi-turn vision-language interactions.

Highlights

  • Superior Performance: A 3B VLM trained with VAGEN outperformed GPT-5, Gemini 2.5 Pro, and Claude 4.5 on five visual agent benchmarks.
  • Modular Design: Allows users to easily plug in custom environments, advantage estimators, and harnesses.
  • Compaction RL: A paradigm that allows training on trajectories longer than the context window by summarizing history.
  • Broad Model Support: Compatible with Qwen2.5-VL, InternVL3.5, and GLM-4.6V-Flash.

Related

  • Project
    Alibaba-NLP/VRAG

    A multi-turn multimodal RAG framework that uses agentic reinforcement learning to enable VLMs to reason and retrieve information across text, images, and videos.

  • Project
    langfengQ/verl-agent

    An extension of veRL for training LLM agents via reinforcement learning, featuring a step-independent rollout mechanism to efficiently handle long-horizon, multi-turn tasks.

  • Project
    mll-lab-nu/RAGEN

    A flexible RL framework for training reasoning LLM agents using multi-turn reinforcement learning and the StarPO algorithm to prevent reasoning collapse.

  • Project
    TsinghuaC3I/MARTI

    An open-source framework for training LLM-based multi-agent systems using reinforcement learning, featuring tree search augmentation for complex reasoning and code generation.

  • Project
    AgentR1/Agent-R1

    A modular framework for Agentic Reinforcement Learning that trains multi-step LLM agents by treating each interaction turn as a step-level MDP transition.