Osilly/Vision-DeepResearch

[ICML 2026 & EMNLP 2026] Multimodal deep-research MLLM and benchmark. The first long-horizon multimodal deep-research MLLM, extending the number of reasoning turns to dozens and the number of search-engine interactions to hundreds.

What it solves

Vision-DeepResearch addresses the limitation of Multimodal Large Language Models (MLLMs) in performing complex, deep research tasks that require iterative visual and textual search. It provides a framework to incentivize deep research capabilities in MLLMs, extending these capabilities from static images to continuous video streams via Video-DeepResearch.

How it works

The project employs a training pipeline consisting of Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to enhance the model's ability to conduct research. It utilizes a data pipeline to generate research-oriented multimodal data and incorporates an "Extract" model to summarize web page contents and a "Judge" model to evaluate performance during RL training. The system can be deployed as an agentic workflow that interacts with search APIs (like SERP and Jina) to gather and synthesize information.

Who it’s for

This project is designed for AI researchers and developers working on multimodal agents, deep research capabilities in LLMs, and the intersection of computer vision and information retrieval.

Highlights

  • Multimodal Research Agents: Creates MLLMs capable of deep research across images and videos.
  • VDR-Bench: A dedicated benchmark for rethinking visual and textual search for MLLMs.
  • Comprehensive Training Suite: Includes SFT and RL code, along with cold-start and RL datasets.
  • Video Extension: Extends deep research capabilities to continuous video streams through Video-DeepResearch.

Related

  • Project
    Alibaba-NLP/VRAG

    A multi-turn multimodal RAG framework that uses agentic reinforcement learning to enable VLMs to reason and retrieve information across text, images, and videos.

  • Project
    langfengQ/verl-agent

    An extension of veRL for training LLM agents via reinforcement learning, featuring a step-independent rollout mechanism to efficiently handle long-horizon, multi-turn tasks.

  • Project
    om-ai-lab/VLM-R1

    VLM-R1 is a framework for training R1-style Large Vision-Language Models using GRPO to improve reasoning and generalization in tasks like visual grounding, math, and object detection.

  • Project
    zai-org/GLM-V

    A series of vision-language models (GLM-4.6V, 4.5V, 4.1V) that enable complex multimodal reasoning, native function calling, and precise visual grounding for agents.

  • Project
    TsinghuaC3I/MARTI

    An open-source framework for training LLM-based multi-agent systems using reinforcement learning, featuring tree search augmentation for complex reasoning and code generation.