Measuring Goodhart's Law in AI Alignment

OpenAI has analyzed the application of Goodhart's Law to AI alignment, demonstrating that when a proxy objective (such as a reward model) becomes the primary target for optimization, it may cease to be a reliable measure of the true objective (such as human intent or factual accuracy).

The Challenge of Proxy Objectives in AI Alignment

Optimizing complex objectives like helpfulness or factual accuracy is difficult because they require costly human verification. To solve this, AI labs use reward models—models trained to predict human preferences—as proxy objectives. However, over-optimizing these proxies can lead to a divergence where the model achieves a high score on the reward model without actually improving the true objective.

Best-of-n Sampling as an Analytical Tool

Best-of-n sampling (also known as rejection sampling or reranking) is a simple method for optimizing a proxy objective by sampling $n$ responses and selecting the one with the highest proxy score. This approach is useful for studying Goodhart's Law because it is mathematically straightforward and provides reliable performance.

Mathematical Framework for Analysis

To measure the impact of optimization, OpenAI utilizes two primary metrics:

True Objective Expectation: $\mathbb{E}{x^{'} \sim P^{'}} [R{\text{true}}(x^{'})]$ measures how well the true objective is being optimized.
KL Divergence: $D_{\text{KL}}(P^{'} \parallel P)$ measures the amount of optimization applied, where $P$ is the original distribution and $P^{'}$ is the optimized distribution.

For best-of-n sampling, the KL divergence has an exact formula for any continuous probability distribution: $\log n - \frac{n-1}{n}$. This allows researchers to precisely track how the true objective varies as the amount of optimization increases.

Efficiency in Estimation

Rather than using a naive Monte Carlo estimator, OpenAI employs a more efficient estimator for the true objective. By considering every possible subset of size $n$ from a larger pool of $N$ samples, they weight each sample by the number of subsets in which it is the best according to the proxy objective. This weighting is based on the binomial coefficient $\binom{N-1}{n-1}$ (specifically using the rank $k$ of the sample).

Empirical Findings from WebGPT

In tests with WebGPT 175B, best-of-n sampling proved competitive with more advanced techniques. Specifically, a best-of-64 model outperformed a reinforcement learning (RL) model, potentially because the larger sample size allowed the model to browse more websites. Even a small increase to best-of-4 provided a significant boost to human preferences.

Scaling Optimization: Best-of-n vs. Reinforcement Learning

While best-of-n sampling is effective for small amounts of optimization, it has limitations at scale because KL divergence grows only logarithmically with $n$.

Computational Efficiency: To reach a KL divergence of 10 nats—a level often reached via RL before Goodhart's Law causes the true objective to decrease—best-of-n sampling would require $n$ to be approximately 60,000.
Information-Theoretic Efficiency: For small KL budgets, best-of-n sampling is more efficient than RL at optimizing both the proxy and true objectives. It is described as a "brute force" approach that is more information-theoretically efficient but less computationally efficient at large KL scales.

Ultimately, the goal of this research is to improve reward modeling and RL practices to reach larger KL divergences without triggering the negative effects of Goodhart's Law.