Stanford CS229 Spring 2026 Lecture 6: Bias-Variance Tradeoff, Regularization, Double Descent, and Hyperband

Bias-Variance Tradeoff: Core Concept and Motivation

The bias-variance decomposition expresses expected test error as the sum of irreducible noise, squared bias, and variance.

The lecture begins by framing the central machine learning question: given a finite noisy sample, how do we pick a model that will generalize? Overfitting occurs when a model is too expressive and fits noise, leading to high variance; underfitting occurs when a model is too simple, leading to high bias. Intuition is built with pictures of fitting a line (underfitting) and a high‑degree polynomial (overfitting) to different draws of the same underlying quadratic function plus Gaussian noise. The line systematically misses the curve (high bias), while the high‑degree polynomial fits each training set perfectly but jumps wildly between sets (high variance). These pictures motivate the formal bias‑variance tradeoff.

Mathematical Derivation of Bias-Variance Decomposition

By expanding the expected squared loss and using the independence of training‑set draw and test‑point noise, the error decomposes into three terms: noise (σ²), bias², and variance.

The derivation fixes a test point X, draws a training set S, learns a hypothesis h(S), then draws a fresh test point (X, Y) where Y = h★(X) + ε with ε ∼ N(0, σ²). The expected loss E[(h(S)(X) − Y)²] is expanded, the cross‑term vanishes because ε has zero mean and is independent of h(S). Adding and subtracting the long‑run average predictor h̄(X) = E_S[h(S)(X)] yields:

E[(h(S)(X) − Y)²] = σ² + E[(h̄(X) − h★(X))²] + E_S[(h(S)(X) − h̄(X))²]

The first term is unavoidable noise, the second is squared bias (how far the average prediction is from the true function), and the third is variance (how much the prediction fluctuates across training sets). This decomposition holds for any hypothesis class and is the basis for analyzing model complexity.

Regularization to Reduce Variance

Ridge regression adds an L2 penalty to the least‑squares objective, trading a small increase in bias for a large reduction in variance.

When the design matrix XᵀX is ill‑conditioned or singular, ordinary least squares can have huge variance because small changes in the data cause large swings in the solution. Ridge regression solves θ̂ = argmin_θ ‖y − Xθ‖² + λ‖θ‖² with λ > 0. The penalty shrinks θ toward zero, which stabilizes the solution. In the eigenbasis of XᵀX, each eigenvalue λ_i is shifted to λ_i + λ, preventing any eigenvalue from becoming zero and thus bounding the variance of θ̂. The bias increases slightly because the solution is pulled away from the unbiased least‑squares estimate, but the variance drops dramatically, especially when some λ_i are very small. This bias‑variance tradeoff is the reason regularization improves test error in practice.

Modern Twists: Double Descent and Robustness

Beyond the classical U‑shaped bias‑variance curve, modern overparameterized models exhibit double descent and unexpected robustness to distribution shifts.

The instructor notes that classical theory predicts test error must rise after the interpolation threshold (where model capacity exceeds the number of training points). However, recent work (e.g., the double descent paper by Misha Belkin et al.) shows that in the heavily overparameterized regime test error can actually decrease again—a phenomenon called double descent. The intuition is that many zero‑loss solutions exist, and implicit bias from the optimizer (e.g., gradient descent) selects a smooth one that generalizes well.

Additionally, robustness to dataset shifts has been examined. The ImageNet V2 paper (discussed with one of its senior authors) reconstructed the ImageNet test set with new images. All models dropped in accuracy by exactly 11 points, but their relative ranking stayed the same. This constant shift suggests that the original test set and the V2 set differ in a systematic way (e.g., weighting or camera characteristics), yet models that performed well on the original also performed well on V2, indicating that adaptive overfitting (leaking test‑set information into training) is not a major concern in these large‑scale vision models.

Model Selection: Cross-Validation and Hyperband

Cross‑validation estimates hyperparameter performance without spoiling the test set; Hyperband allocates compute efficiently to promising configurations.

To choose hyperparameters (e.g., the ridge penalty λ) we need a validation signal that does not contaminate the final test set. A hold‑out (dev) set serves this purpose, but it wastes data. K‑fold cross‑validation splits the training data into K folds, trains on K−1 folds and validates on the held‑out fold, rotating which fold is held out. This provides a low‑bias estimate of generalization while using all data for training at some point.

When training each candidate model is expensive, Hyperband improves efficiency. It evaluates all configurations for a small number of steps, discards the worst half, doubles the training time for the survivors, and repeats until one configuration remains. Each round spends roughly the same total compute, but better‑performing configurations receive exponentially more resources. The method is simple to implement, has strong theoretical guarantees under certain monotonicity assumptions, and reduces the number of full‑training runs needed for hyperparameter search.

These tools—bias‑variance analysis, regularization, modern insights like double descent, and practical model‑selection algorithms—form a cohesive framework for building models that generalize from finite, noisy data.