Policy Gradient, PPO Clipping, and Chain‑of‑Thought RL for LLMs – Stanford CS229 Lecture 20

Policy Gradient Derivation and Baseline

The policy gradient theorem states that the gradient of the expected return equals the expectation of the sum over time of the gradient of the log probability of actions multiplied by the return. Because the log‑probability gradient has zero expectation when no reward is present, any term that does not depend on the action can be added or subtracted without changing the expectation. This property allows the introduction of a baseline that depends only on the state; subtracting such a baseline leaves the expected gradient unchanged but can reduce variance in practice.

Importance Sampling and the On‑Policy Limitation

The naive policy gradient estimator requires sampling trajectories from the current policy, making it on‑policy: after a parameter update the old samples cannot be reused. Importance sampling corrects this by reweighting samples from an old policy π_old with the ratio π_θ(a|s) / π_old(a|s). In practice only the action part of the ratio is tractable, leading to an estimator that uses the old policy for states but corrects the action distribution via the ratio.

Proximal Policy Optimization (PPO) Clipping Rules

PPO modifies the surrogate objective by clipping the probability ratio to prevent overly large updates. For a given advantage Â_t:

  • If Â_t > 0 and the ratio r_t = π_θ(a_t|s_t) / π_old(a_t|s_t) is above 1 + ε_high, the contribution is clipped to (1 + ε_high) Â_t, yielding zero gradient.
  • If Â_t > 0 and r_t is below the clipping threshold, the term is r_t Â_t and a gradient is taken.
  • If Â_t < 0 and r_t is below 1 − ε_low, the contribution is clipped to (1 − ε_low) Â_t, again giving zero gradient.
  • If Â_t < 0 and r_t is above the lower threshold, the term is r_t Â_t and a gradient is taken. Typical values mentioned in the lecture are ε_high ≈ 0.28 and ε_low ≈ 0.2. The clipping implements the idea that no further update is needed when the new policy is already sufficiently better or worse than the old policy.

Variants of PPO (GRPO and SIPO)

The lecture describes two extensions of the basic PPO scheme.

  • GRPO (referred to as an “advanced version of PO”) applies the same clipping logic but, when the ratio exceeds the upper threshold, it sets the contribution to zero rather than keeping a constant clipped value.
  • SIPO (called “SIS pole” in the transcript) differs by keeping a non‑zero, constant gradient when the ratio is large: it clips the ratio to 1 instead of zeroing the term, thereby preserving some learning signal while still limiting magnitude. Both variants aim to trade off stability and learning speed, with SIPO retaining more gradient signal in the high‑ratio regime.

Applying RL to LLMs for Chain‑of‑Thought

To train a language model to produce chain‑of‑thought reasoning, the generation process is treated as a Markov decision process where states are the concatenation of prompt and previously generated tokens, actions are the next token, and transitions are deterministic (appending the chosen token). The reward is given only at the end of a trajectory and is based on whether the final answer matches a ground‑truth solution; intermediate thinking tokens are not directly rewarded. This setup allows the use of any policy gradient method, including PPO or its variants, to optimize the model for correct answers while encouraging multi‑step reasoning.

Reward Design and Baselines in LLM Training

Because the reward is binary (1 for a correct answer, 0 otherwise), variance can be high. A common baseline is the average reward over several sampled trajectories for the same prompt: compute rewards R₁…R₈ for eight rollouts, take their mean R̄, and subtract R̄ from each individual reward to obtain an advantage estimate. This baseline depends only on the prompt (state) and thus satisfies the condition that subtracting it does not change the expected gradient. Additional normalisation (e.g., dividing by standard deviation) may be applied but was noted as optional for further study.

Sources