ThinkingBox-Training: RL fine-tuning of Qwen3.8-27B with database-state rewards

TL;DR

ThinkingBox-Training (released on GitHub) turns executable checks of a task’s final database state into binary RL rewards and uses them to fine‑tune Qwen3.8‑27B, boosting the share of benchmark tasks that succeed on all 20 attempts from 7.5% to 21.1% and reducing the estimated cost per dependable task to $8.00.

What ThinkingBox‑Training Does

ThinkingBox was built to grade agents by comparing the terminal database state with the expected outcome. The authors realized that the same executable check can serve as a reward signal for reinforcement‑learning (RL) training on tasks that are outside the original ThinkingBox‑Bench benchmark.

  • The reward is binary: 1 if the final database state satisfies the deterministic check, otherwise 0.
  • A minority of tasks use an LLM judge for narrow rubric questions, but the vast majority rely on the deterministic check.
  • The training loop runs full‑parameter RL fine‑tuning (GRPO) on a curated pool of tasks whose difficulty is policy‑relative—i.e., tasks that the base model sometimes succeeds and sometimes fails.

Offline Policy‑Relative Difficulty Selection

Only tasks that can produce mixed reward outcomes are useful for GRPO. The authors therefore:

  1. Probe each candidate task n times with the base policy (the un‑tuned Qwen3.8‑27B).
  2. Model the success probability with a Jeffreys‑posterior Beta distribution: Beta(k+0.5, n‑k+0.5) where k is the number of successes.
  3. Bucket the posterior mass into five intervals (0‑0.08, 0.08‑0.40, 0.40‑0.60, 0.60‑0.90, 0.90‑1).
  4. Keep tasks in the three middle buckets (difficult, intermediate, frequently solved) and discard tasks that are almost always solved or almost always failed.
  5. The resulting fixed pool contained 187 tasks for the reported run.

"Difficulty is policy‑relative: a task is hard for the current model only until that model learns it."

Training Loop and Infrastructure

The stack runs on 24 H100 GPUs (3 nodes × 8 GPUs) and follows six stages (Figure 5 in the post):

  1. Candidate task ingestion.
  2. Offline probing and difficulty filtering.
  3. Rollout generation (8 episodes per group).
  4. Terminal‑state grading.
  5. GRPO reward computation (centering within each group).
  6. Parameter update.

Key implementation details:

  • GRPO uses token‑mean advantage with a learning rate of 1e‑6 and zero weight decay.
  • Zero‑variance groups are replaced using a DAPO‑style oversampling strategy.
  • All parsing (tool calls, arguments, reasoning extraction) is shared between training and evaluation via trainer/core/tb_interpret.py to guarantee reward‑evaluation parity.

Quantitative Impact on Qwen3.8‑27B

The model was evaluated on the out‑of‑distribution ThinkingBox‑Bench (507 tasks, 20 repetitions each). Table 1 in the post reports:

Metric Base Model RL‑Fine‑Tuned Δ
Observed 20/20 (tasks that pass every attempt) 38 (7.50%) 107 (21.10%) +69
pass@1 (average success) 5,242 (51.70%) 6,163 (60.78%) +9.08 pts
pass@20 (at‑least‑once coverage) 453 (89.35%) 446 (87.97%) ‑7

The RL fine‑tuning sharpened reliability: repeatable success rose sharply while at‑least‑once coverage fell slightly.

Domain‑Level Effects

Pass@1 and observed 20/20 improved across all five benchmark domains (Retail, Auto insurance, Travel, Neobank, Consulting). Pass@20 increased in Neobank and Consulting but decreased in Retail, Travel, and Auto insurance. Retail saw the largest observed‑20/20 gain (+24.5 points).

Cost Efficiency

Using the same token‑usage estimates as the first post, the fine‑tuned checkpoint reduced evaluation cost and per‑task cost:

Measure Base RL‑Fine‑Tuned Δ
Campaign cost (507 × 20 runs) $925.80 $855.58 ‑7.6 %
Cost per successful attempt $0.177 $0.139 ‑21.4 %
Tasks passing 20/20 38 107 2.8×
Cost per dependable task $24.36 $8.00 ‑67.2 %

At $8.00 per dependable task, the fine‑tuned model approaches the cost efficiency of GPT‑5.4, GPT‑6 Astra, and Claude Opus 5.5.

Running the Stack Yourself

ThinkingBox‑Training is open‑source at github.com/microsoft/thinkingbox-training. The repository includes:

  • Data preparation (docs/data-preparation.md) – task hydration, policy probing, difficulty filtering, benchmark exclusion.
  • Reference training configuration (docs/reference-q38-run.md) – a validated 24‑GPU run for full‑parameter Qwen3.8‑27B.
  • Evaluation guide (docs/evaluation.md) – merging the checkpoint, serving with vLLM, and running the full 507 × 20 benchmark via OpenEnv.

The post provides a step‑by‑step Linux/WSL setup (Python 3.12, uv, NVIDIA CUDA/NCCL), environment creation, package installation, and launch scripts for Ray‑based distributed training.

Takeaways for Practitioners

  1. Probe before you train. Estimate the policy‑relative difficulty distribution of your task pool; keep only tasks that yield mixed success.
  2. Maintain parity between training and evaluation parsers. Any mismatch in tool‑call extraction will reward the wrong behavior.
  3. Report both coverage and repeatability. Pass@1 alone hides reliability; include observed‑20/20 (or similar) to convey dependable performance.

The authors stress that the core contribution is the training loop and data curation, not the single checkpoint. The approach should generalize to any agent that manipulates persistent state (databases, file systems, etc.).

References & Resources

The code is MIT‑licensed; the benchmark data is CDLA‑Permissive‑2.0.

Sources