Gaussian Discriminant Analysis (GDA) – Stanford CS229 Spring 2026 Lecture 5

Generative vs Discriminative Modeling

Generative models learn the joint distribution P(X, Y) or equivalently P(X|Y) together with the class prior P(Y), whereas discriminative models learn the conditional P(Y|X) directly. In this lecture Gaussian Discriminant Analysis (GDA) is presented as the first generative algorithm studied in the course, contrasting with the discriminative models (linear regression, logistic regression) seen earlier.

Gaussian Distributions in 1D and 2D

A multivariate Gaussian in R^n is characterized by a mean vector μ and a covariance matrix Σ. The covariance must be symmetric and positive‑definite, which guarantees that Σ has a well‑defined inverse and that its eigenvalues are all positive. The density involves the term (x−μ)^T Σ^−1 (x−μ) inside an exponential, and the normalizing constant contains the determinant of Σ. Contours of equal probability are ellipses whose orientation and shape are determined by Σ.

Gaussian Discriminant Analysis Model

GDA assumes that for each class k the features X follow a Gaussian distribution with class‑specific mean μ_k but a shared covariance Σ across all classes. The class label Y is modeled as a Bernoulli variable with prior π_k = P(Y=k). Thus the generative story is: first draw a label according to the class priors, then draw X from the Gaussian corresponding to that label.

Maximum Likelihood Estimation for GDA

Given a labeled dataset, the maximum likelihood estimates have closed‑form solutions. The class prior is estimated by the empirical frequency: π̂k = N_k / N where N_k is the number of examples with label k. The class‑specific mean is the average of the features assigned to that class: μ̂_k = (1/N_k) Σ{i:y_i=k} x_i. The shared covariance is estimated by pooling the scatter of all classes: Σ̂ = (1/N) Σ_{k} Σ_{i:y_i=k} (x_i − μ̂_k)(x_i − μ̂_k)^T. These formulas involve only counting and averaging, requiring no iterative optimization.

Decision Boundary and Relation to Logistic Regression

When the covariance Σ is shared, the log‑likelihood ratio between two classes reduces to an affine function of x, so the GDA decision boundary is a hyperplane (linear). If each class is allowed its own covariance, the boundary becomes quadratic (quadratic discriminant analysis, QDA). The lecture notes that this linear boundary can also be obtained by a logistic regression model, showing that many generative assumptions on the data lead to the same discriminative form.

Naive Bayes for Discrete Features

For binary word‑presence features, naive Bayes assumes that the features are conditionally independent given the class. This assumption reduces the number of parameters from exponential in the number of features to linear: one Bernoulli parameter per feature per class plus the class priors. The maximum likelihood estimates are simply the empirical frequencies of each word within each class (with Laplace smoothing to avoid zero probabilities). The resulting classifier is cheap to train and evaluate, and it serves as a simple generative baseline for tasks such as spam filtering.