rlresearch/dr-tulu

DR Tulu – Reinforcement‑Learning Agents for Deep Research

What it is – DR Tulu is an open‑source research‑grade language model (8 B parameters) and accompanying code for training and running deep‑research agents. The agents can browse the web, query scholarly APIs, and synthesize long‑form answers to scientific questions. The repository bundles three parts:

Component	Purpose
agent/	A library (dr‑agent‑lib) that implements a MCP‑based tool backend, high‑concurrency async request handling, and a flexible prompting interface. It also contains evaluation scripts for benchmarking the agents.
rl/open‑instruct/	Reinforcement‑learning training code (built on AllenAI’s Open‑Instruct) that uses GRPO (a policy‑gradient algorithm) together with evolving rubrics to teach agents how to search, retrieve, and reason.
sft/llama‑factory/	Supervised‑fine‑tuning pipelines (based on LLaMA‑Factory) for turning raw demonstration data into a research‑capable model before RL.

---

Quick‑start: Interactive demo (CLI)

1. Create a conda env and install the agent library:

   cd agent/
   conda create -n dr_agent python=3.10 -y && conda activate dr_agent
   uv pip install -e .
   

2. Set API keys for the external tools the agent can call (Serper, Semantic Scholar, Jina Reader).
3. Launch the chat (requires 1‑2 GPUs):

   uv run --extra vllm python scripts/launch_chat.py --model rl-research/DR-Tulu-8B
   

   The script automatically starts a VLLM server for the model and an MCP server that routes tool calls. You can then ask research‑style questions and watch the agent browse, retrieve papers, and generate a synthesized answer.

---

Running evaluations

The repo provides a full evaluation harness for benchmarks such as HealthBench, Deep Research Bench, SimpleQA, Genetic Diseases, 2Wiki, and WebWalker.

# Start two VLLM model servers (DR‑Tulu and a comparison model)
CUDA_VISIBLE_DEVICES=0 vllm serve rl-research/DR-Tulu-8B --dtype auto --port 30001 --max-model-len 40960
CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-8B --dtype auto --port 30002 --max-model-len 40960

# Start the MCP tool‑backend
python -m dr_agent.mcp_backend.main --port 8000

After the services are up, a loop over the desired tasks runs the agent to generate answers and then calls scripts/evaluate.py to compute benchmark scores. Detailed per‑benchmark instructions live in agent/evaluation/README.md.

---

Training pipelines

• Supervised fine‑tuning (SFT) – Use the sft/llama-factory/ directory. It follows the standard LLaMA‑Factory workflow: prepare a JSONL of instruction‑response pairs, configure a YAML, and launch training on one or more GPUs.
• Reinforcement learning (RL) – Use rl/open-instruct/. The code trains a reward model, then runs GRPO with evolving rubrics that progressively tighten the evaluation criteria, encouraging the policy to improve its search‑and‑synthesis behavior. Both sections point to their own README files for exact command‑line flags, data‑format specifications, and hyper‑parameter recommendations.

---

Who built it?

The project is from Allen Institute for AI (AI2), in collaboration with students at the University of Washington, Carnegie Mellon University, and MIT. The associated paper (arXiv 2511.19399) describes the methodology and reports that DR Tulu‑8B matches OpenAI’s Deep Research model on long‑form research benchmarks.

---

How to cite

@article{shao2025dr,
  title={DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research},
  author={Shao, Rulin and Asai, Akari and Shen, Shannon Zejiang and Ivison, Hamish and Kishore, Varsha and Zhuo, Jingming and Zhao, Xinran and Park, Molly and Finlayson, Samuel G and Sontag, David and others},
  journal={arXiv preprint arXiv:2511.19399},
  year={2025}
}

---

TL;DR

DR Tulu is a research‑grade, 8 B‑parameter language model plus a full stack for training (SFT + RL) and deploying agents that can browse the web, query scholarly APIs, and produce long‑form, citation‑rich answers. The repo is organized into a reusable agent library, RL training code, and SFT scripts, and includes ready‑to‑run evaluation pipelines and an interactive CLI demo.