verl-project/verl-recipe

What is verl‑recipe?

verl‑recipe is a companion repository that bundles recipes – ready‑made example projects and configurations – for the VeRL library (found at https://github.com/verl-project/verl). VeRL is an open‑source framework for reinforcement‑learning (RL) research with large language models (LLMs). Each recipe demonstrates a specific RL technique, benchmark, or integration (e.g., tool‑use with LLMs, single‑stream policy optimisation, multi‑turn math solving with LangGraph, etc.).

How it works

1. Submodule integration – The repo is meant to be added as a submodule inside the main VeRL checkout (verl/recipe). This keeps the recipes version‑locked to the VeRL version they were tested against.
2. Version pinning – Every recipe directory contains a REQUIRED_VERL.txt file that records:
   • The upstream VeRL git URL.
   • Whether the recipe follows the current main branch or pins a specific commit / release tag.
   • The exact pip install … command needed to install the matching VeRL version.
   • For rolling recipes, three commit hashes are stored so both the core library and the recipe tree are reproducibly pinned.
3. One‑shot installer – The install_verl.sh script reads a recipe’s REQUIRED_VERL.txt and automatically runs the appropriate pip install (or a git‑based editable install). It can list all recipes, show the exact command it would run, or install a chosen recipe with optional flags (--method git, --option …, etc.).
4. Contribution workflow – When adding or updating a recipe, contributors must update the REQUIRED_VERL.txt to reflect the new VeRL version. The repo uses pre‑commit with the ruff linter to keep code tidy.

What you can do with it

• Run state‑of‑the‑art RL experiments on LLMs without building the setup from scratch. For example:
  • retool – RL for strategic tool use in LLMs.
  • langgraph_agent – A tiny LangGraph ReactAgent that rolls out multi‑turn math expressions.
  • spo – Implementation of Single‑stream Policy Optimization (arXiv:2509.13232).
  • partial_rollout – Synchronous RL with rollout interruption/resume to reduce GPU idle time (APRIL‑style).
  • verl_tinker – An HTTP server exposing VeRL actors, usable with the Tinker cookbook client.
• Experiment with research variants by selecting options defined in a recipe (e.g., different paper‑specific checkpoints for DAPO or FlowRL).
• Reproduce published results because each recipe pins the exact VeRL commit and any auxiliary submodule commits it depends on.

Who is it for?

• Researchers and engineers working on reinforcement learning for LLMs who want reproducible, community‑vetted starting points.
• Students looking for concrete examples of advanced RL methods applied to language models.
• Developers who already use VeRL and need a quick way to add new experimental setups.

Getting started (quick steps)

# Clone VeRL and pull in the recipes as a submodule
git clone https://github.com/verl-project/verl.git
cd verl
git submodule update --init --recursive recipe

# List all available recipes and their pinned install commands
./recipe/install_verl.sh --list

# Install a specific recipe, e.g., the "retool" RL tool‑use example
./recipe/install_verl.sh --recipe retool

The script will display the exact pip install line it will execute, and you can add --show for a dry‑run or --yes to skip confirmation.

---

In short: verl‑recipe is a curated collection of reproducible RL‑for‑LLM examples that live alongside the VeRL library, with tooling to keep the library and recipes perfectly in sync.