The N Implementation Details of RLHF with PPO – Hugging Face Blog Summary
The Hugging Face blog post reproduces OpenAI’s 2019 RLHF codebase, matches its learning curves for stylistic tasks, and enumerates implementation details that affect reproducibility.
Matching Learning Curves
The reproduction yields learning curves nearly identical to OpenAI’s original codebase for sentiment and descriptiveness tasks. The authors ran the original TensorFlow 1.x repository on AWS p3dn.24xlarge instances to obtain baseline metrics, then used those metrics to validate their PyTorch‑based reproduction.
General Implementation Details
The reward model and policy value head receive the concatenated query and response as input. Sequences are padded to a fixed length with a special padding token and truncated when too long, and position indices are adjusted for padding tokens during logit calculation. Response generation samples a fixed‑length output without stopping at EOS tokens, and learning rates are annealed to zero for both reward model and policy training. Different random seeds are used per GPU process to encourage exploration.
Reward Model Implementation Details
The reward model outputs only the value at the last token of the concatenated query‑response pair. The reward head weight is initialized from a normal distribution with variance 1/(sqrt(d_model)+1) and bias set to zero. Reward normalization is performed before and after training by computing empirical mean and std of rewards from a fixed reference policy and scaling to unit variance and zero mean.
Policy Training Implementation Details
Logits are divided by the sampling temperature before computing log probabilities. The value head weight is initialized to zero mean and zero variance. Dropout is disabled during policy training. Rejection sampling enforces a period between tokens 16 and 24, assigning a fixed low reward otherwise. The discount factor γ is set to 1. Training uses batches, minibatches, and micro‑batches for gradient accumulation. A per‑token KL penalty is added to rewards, and per‑minibatch reward and advantage whitening is applied (with mean shifting for advantages only). The value function is clipped as in standard PPO, and the KL penalty coefficient β is adapted based on observed KL divergence.
PyTorch Adam Optimizer Numerical Issues w.r.t RLHF
PyTorch’s Adam implementation differs from TensorFlow’s in the placement of epsilon inside the denominator, leading to a smaller effective normalization term early in training and thus more aggressive gradient updates. Experiments show that PyTorch Adam yields higher log‑probability variance, larger ratio extremes, and increased approximate KL and clip fraction compared to TensorFlow‑style Adam, especially with larger models such as gpt2‑xl.
Limitations
The work does not attempt to reproduce the summarization tasks from CNN/DM or TL;DR due to high computational cost and training instability. The original code requires 8× V100 32GB GPUs, and the reproduction suffered from low GPU utilization (~30%), making runs expensive and brittle.
Conclusion
By reproducing OpenAI’s RLHF codebase and documenting its implementation details, the authors provide a reference for understanding RLHF engineering practices and highlight a subtle optimizer difference that can significantly affect training dynamics.