TIGER-AI-Lab/Pixel-Reasoner

Pixel-Level Reasoning Model trained with RL [NeuIPS25]

Pixel Reasoner – Vision‑Language Models that think in pixel space

What it is – A research‑grade framework that equips a Vision‑Language Model (VLM) with explicit visual‑manipulation operations (e.g., zoom‑in, select‑frame) so the model can actively explore images or video frames while reasoning. The authors train a 7‑B VLM in two stages:

  1. Instruction‑tuning on synthetic multi‑turn traces that teach the new pixel‑space actions.
  2. Curiosity‑driven reinforcement learning that rewards the model for balancing textual reasoning with useful visual operations.

The result is a VLM that can query visual evidence on‑the‑fly, achieving state‑of‑the‑art scores on several visual‑reasoning benchmarks (V* bench 84 %, TallyQA‑Complex 74 %, InfographicsVQA 84 %).

Key capabilities

  • Pixel‑space operations – built‑in commands such as zoom-in, select-frame that the model can invoke during generation.
  • Two‑phase training – instruction‑tuning (adapted from Open‑R1) → curiosity‑driven RL (adapted from VL‑Rethinker).
  • Multi‑turn trajectories – both the SFT and RL pipelines accept sequences of visual‑textual interactions.
  • Mixed image/video data – the RL stage can train on video frames as well as static images.
  • vLLM‑based inference – fast batched generation for evaluation and downstream use.

Getting started (quick‑start)

  1. Clone the repo and navigate to the desired sub‑folder (instruction_tuning or curiosity_driven_rl).
  2. Install dependencies – follow the provided install.md (or installation.md) which sets up the Python environment, vLLM, and openrlhf.
  3. Instruction tuning
    cd instruction_tuning
# edit sft.sh to point to your model checkpoint and the SFT dataset
bash sft.sh
  4. Curiosity‑driven RL – prepare the RL dataset, download the warm‑start checkpoint (PixelReasoner‑WarmStart), then run the multi‑node or single‑node script:
    cd curiosity_driven_rl
export policy=/path/to/PixelReasoner‑WarmStart/checkpoint-246
bash scripts/train_vlm_multi.sh   # or train_vlm_single.sh
    (All hyper‑parameters are exposed as environment variables; the README lists the most important ones such as MAX_PIXELS, lr, bsz, etc.)
  5. Evaluation – use the provided HF collections for image‑based (VStar) and video‑based (MVBench) benchmarks. Example for VStar:
    export benchmark=vstar
export policy=/path/to/trained/model
bash scripts/eval_vlm_new.sh
    Adjust eval_bsz, MAX_PIXELS, and GPU counts as needed.

Models & data (publicly hosted)

Item Hugging Face link What it contains
PixelReasoner‑RL‑v1 TIGER-Lab/PixelReasoner-RL-v1 Trained 7 B checkpoint ready for inference/evaluation
PixelReasoner‑WarmStart TIGER-Lab/PixelReasoner-WarmStart Pre‑RL checkpoint used as the RL policy initializer
SFT data TIGER-Lab/PixelReasoner-SFT-Data Multi‑turn instruction‑tuning trajectories
RL data TIGER-Lab/PixelReasoner-RL-Data Queries and reward signals for curiosity‑driven RL
Eval collections HF collection linked in README Parquet files for VStar, MVBench, and other visual‑reasoning benchmarks

Demo & website

  • Online demo – a Hugging Face Space where you can interact with the model (https://huggingface.co/spaces/TIGER-Lab/Pixel-Reasoner).
  • Project website – contains paper PDF, model cards, and visual examples (https://tiger-ai-lab.github.io/Pixel-Reasoner/).

Common pitfalls (as noted in the README)

  • Context‑length overflow – keep MAX_PIXELS low enough that image tokens + text tokens stay under the model’s 10 240 token limit.
  • Transformer/vLLM version mismatch – reinstall the exact transformer commit the repo pins if you see dtype errors.
  • Log‑prob batch size – logp_bsz must be 1 (or --micro_rollout_batch_size=1) for correct PPO log‑prob computation.
  • Distributed env vars – when using Ray, propagate MAX_PIXELS/MIN_PIXELS via RUNTIME_ENV_JSON.

Contact & citation

  • Lead contact: Haozhe Wang (jasper.whz@outlook.com) – bug help for RL.
  • SFT contact: Muze (dlwlrma314516@gmail.com) – SFT and alternative RL code.
  • Citation:
    @article{pixelreasoner,
  title={Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning},
  author={Su, Alex and Wang, Haozhe and Ren, Weiming and Lin, Fangzhen and Chen, Wenhu},
  journal={arXiv preprint arXiv:2505.15966},
  year={2025}
}

Bottom line

Pixel Reasoner is a full‑stack research project that introduces pixel‑space actions for VLMs and provides the code, data, and pretrained checkpoints needed to reproduce the results or build your own visual‑reasoning agents. It is firmly within the frontier AI/ML domain.

Related

  • Dispatch
    Vision Language Models Explained

    Hugging Face provides a comprehensive guide to Vision Language Models (VLMs), detailing their architecture, open-source options, evaluation benchmarks, and new experimental support for fine-tuning via the TRL library.

  • Dispatch
    Hugging Face Vision Language Models 2025 Update

    Hugging Face provides a comprehensive overview of the 2024-2025 evolution of Vision Language Models (VLMs), highlighting trends in any-to-any architectures, reasoning models, and the rise of Vision-Language-Action (VLA) models for robotics.

  • Project
    alexzhang13/rlm

    An inference engine and training environment for Recursive Language Models (RLMs) that enable LLMs to handle near-infinite context by programmatically decomposing tasks via recursive sub-calls in a REPL environment.

  • Dispatch
    TRL adds Direct Preference Optimization support for Vision‑Language Models

    Hugging Face added Direct Preference Optimization (DPO) support for Vision‑Language Models in the TRL library, enabling fine‑tuning of models like Idefics‑2 with preference data using bfloat16 quantization and LoRA to fit on a single GPU.