Stanford CS229 Spring 2026 Lecture 4: Exponential Family, GLMs, and Softmax

The exponential family provides a single functional form that captures many common distributions, allowing inference and learning to be carried out with the same derivations regardless of the specific distribution. This matters because it underlies widely used models such as logistic regression, softmax multiclass classification, and the final layer of modern neural networks.

Exponential Family Form and Motivation

A probability distribution belongs to the exponential family if it can be written as exp(ada^T * t(y) - a(ada) + b(y)), where ada is the natural parameter, t(y) is the sufficient statistic, a(ada) is the log partition function, and b(y) is the base measure that does not depend on ada. This form ensures that normalization is handled by a(ada), which makes the distribution sum to one.

Examples: Bernoulli and Gaussian

For a Bernoulli variable y ∈ {0,1} with mean φ, the sufficient statistic t(y) is y, the natural parameter ada is log(φ/(1−φ)), the log partition function a(ada) is log(1+exp(ada)), and b(y) is zero. For a Gaussian with known variance, t(y) is y, ada is μ/σ², a(ada) is μ²/(2σ²) plus a constant, and b(y) contains the quadratic term −y²/(2σ²). Both examples fit the prescribed form.

Key Properties: Log Partition Function Gives Expectation and Variance

The first derivative of the log partition function a(ada) with respect to ada equals the expectation of the sufficient statistic t(y). The second derivative equals the variance of t(y). This holds for any distribution in the exponential family, so once the log partition function is known, expectation and variance are available "for free" without re‑deriving moments.

Generalized Linear Models (GLM) Framework

In supervised learning we model p(y|x;θ) by choosing an exponential family distribution whose natural parameter ada is set equal to θ^T x. The predicted output h_θ(x) is the expectation of t(y) given x, which is obtained by differentiating a(ada). Learning proceeds by maximizing the log likelihood of the data, which leads to a gradient ascent update that can be written as θ ← θ + (y − h_θ(x)) x when the sufficient statistic is the identity.

Softmax and Multiclass Classification

When y is a discrete class label encoded as a one‑hot vector, the sufficient statistic is y itself and the natural parameter ada is a vector of scores θ_j^T x for each class j. The log partition function a(ada) is log(∑_j exp(θ_j^T x)). The resulting distribution is the softmax: p(y=j|x) = exp(θ_j^T x) / ∑_k exp(θ_k^T x). This is exactly the multiclass classification model used in neural networks.

Connection to Logistic Regression (K=2 case)

With two classes the softmax reduces to logistic regression. Because the probabilities must sum to one, only one score is free; defining θ = θ_1 − θ_2 collapses the two‑class softmax to σ(θ^T x) = 1/(1+exp(−θ^T x)), which is the logistic function.

Training: Maximum Likelihood, Cross Entropy, Gradient Descent

Maximizing the log likelihood of the softmax model yields the cross‑entropy loss: −∑_j y_j log(p_j), where y_j is the one‑hot label. Stochastic gradient descent on this loss updates parameters by θ ← θ + (y − p) x, where p is the vector of softmax probabilities. This update rule matches the intuitive idea of increasing the score for the correct class and decreasing scores for incorrect classes.

Label Smoothing Intuition

Label smoothing replaces the hard one‑hot target with a distribution that puts a small amount of probability on incorrect classes. This prevents the model from becoming overconfident, acts as a regularizer, and makes training more robust to noisy labels. The same cross‑entropy formula works with the softened targets.

Practical Notes: Implementation and Numerical Stability

In practice libraries compute softmax in a numerically stable way by subtracting the maximum score before exponentiation. The underlying mathematics remains the same: exponentiate the scores, normalize by their sum, and obtain a probability distribution that sums to one. These operations are the final layer in most modern AI systems, including those that generate text token by token.