Optimizing 0.6B Dense Model RL on TPU v7 via vLLM

Executive Summary

Researchers have optimized the reinforcement learning (RL) training cycle for 0.6B dense models on Google Cloud TPU v7, reducing the full rollout-to-update cycle from approximately 100 minutes to 56 seconds. This 107x speedup was achieved by addressing system-level bottlenecks—specifically decode scheduling, synchronization, attention geometry, and KV-cache lifecycle—rather than focusing solely on FLOPs. All optimizations have been upstreamed to the vLLM, Tunix, and MaxText repositories, allowing developers to reproduce these results using the provided tpu-recipes configuration.

The Challenge of Small Dense Models

For large dense or mixture-of-experts (MoE) models, distributed training is typically dominated by matrix arithmetic and communication overheads. However, when training a 0.6B parameter model, the computational work per layer is significantly reduced, causing fixed system costs to become the primary bottleneck. In this regime, host-side dispatch, synchronization barriers, KV-cache management, and kernel launch overheads consume a disproportionate share of wall-clock time. Without co-design, these overheads prevent the TPU v7 MXUs from maintaining high utilization.

System Co-Design and Optimization

The optimization campaign followed a three-phase approach to align the software stack with the workload's specific geometry:

Phase I: Baseline Configuration

Initial un-tuned runs generated approximately 4K tokens/s. By performing mesh sweeps, rebalancing data and tensor parallelism, and tuning HBM limits, the team increased throughput to 30K tokens/s and reduced the cycle time from 100 minutes to 21 minutes.

Phase II: Decode Co-Design

To overcome host-side dispatch limitations, the team implemented "Continue Decode" in vLLM, which compiles multiple decode iterations into a single XLA device loop. Furthermore, they refactored the Ragged Paged Attention (RPA v3) kernel to decouple HBM fetch geometry from VMEM compute tiling. This allowed for better overlap between DMA transfers and TensorCore execution, lifting decode throughput to 130.5K tokens/s.

Phase III: Full-Cycle Optimization

Final gains were realized by optimizing the entire RL loop. Key improvements included:

  • KV Persistence: Decoupling logical cache invalidation from physical HBM allocation to avoid unnecessary reallocations during weight updates.
  • Continuous Batching: Keeping the fused decode loop active by retiring finished sequences as masked slots.
  • Training Retuning: Adjusting Splash Attention block sizes to match the production sequence geometry.
  • Parallelism Shift: Moving to a TP1×DP128 configuration to maximize rollout concurrency.

Benchmark Methodology

The team utilized a 64-chip Google Cloud TPU v7 slice running Qwen3-0.6B with Group Relative Policy Optimization (GRPO). The workload maintained a strict 8,192-token response cap across 2,048 concurrent completions. All reported metrics reflect steady-state wall-clock time, excluding compilation and warmup, with the final configuration sustaining ~183K tokens/s.

Negative Results and Lessons Learned

Several hypotheses were tested and discarded, saving significant engineering effort:

  • Weight/KV Quantization: Neither INT8 weights nor FP8 KV-caches provided meaningful speedups, as the bottleneck was driven by fixed-cost execution rather than memory bandwidth.
  • Shared-Prefix Attention: Composed prefix/suffix attention showed no net gain over the optimized production RPA kernel at scale.
  • Training Microbatching: Increasing training microbatch sizes did not reduce total FLOPs or shorten the training phase.

Reproducibility

All changes are merged into the open-source stack. Users can reproduce the 56-second cycle using the tpu-recipes repository. The final configuration requires specific environment variables, such as RPA_V3_DECODE_BLOCK_SIZES="1,2048,1,2048" and CONTINUE_DECODE_EOS_CHECK_INTERVAL="0", to ensure the fused XLA loop operates without unnecessary synchronization.

Sources