Stanford CS229 Machine Learning Spring 2026 Lecture 2: Supervised Learning Setup
Supervised Learning Setup
Supervised learning learns a hypothesis from labeled training pairs to predict outputs for new inputs. A hypothesis is a function h that maps an input space X to an output space Y. The training set consists of pairs (x^{(i)}, y^{(i)}) where x^{(i)} ∈ X is an input example and y^{(i)} ∈ Y is its label. The goal is to choose h so that its predictions on unseen inputs are close to the true outputs, a property called generalization.
Regression vs Classification
When the output space Y contains real numbers the problem is regression; when Y contains a finite set of discrete labels the problem is classification. Regression examples include predicting house prices; classification examples include detecting whether an image contains a cat or a dog. The lecture notes that chat‑GPT uses a classifier head to guess the next word, illustrating a classification component inside a larger system.
Linear Hypothesis and Least Squares
A linear (affine) hypothesis takes the form h_θ(x) = θ^T x, where θ is a vector of parameters and the convention x_0 = 1 absorbs the intercept term. The parameters are chosen by minimizing the empirical risk measured by squared error. The loss function is J(θ) = ½ Σ_{i=1}^n (h_θ(x^{(i)}) - y^{(i)})^2. Setting the gradient of J with respect to θ to zero yields the normal equations, which give a closed‑form solution when the design matrix satisfies certain conditions.
Gradient Descent Basics
Gradient descent finds a parameter vector that minimizes J by iteratively updating θ in the direction opposite the gradient: θ ← θ - α ∇J(θ). The step size α controls how far each update moves. If α is too large the updates can overshoot the minimum and cause oscillation; if α is too small convergence is slow. The lecture notes that choosing α is a practical concern and that adaptive optimizers (e.g., Adam, Adagrad) can adjust it automatically.
Stochastic Gradient Descent and Mini‑Batching
Stochastic gradient descent (SGD) replaces the full‑batch gradient with an estimate computed on a randomly selected mini‑batch of training examples. This reduces the per‑iteration cost from O(n) to O(batch size) and enables training on very large datasets. The mini‑batch is drawn uniformly at random; sampling with or without replacement yields similar behavior in practice, and without replacement is often easier to implement. The batch must be representative of the overall data; otherwise the model may overfit to the sampled subset (e.g., seeing only cats and then only dogs). Batch size involves a trade‑off: smaller batches give noisy gradients but more frequent updates, while larger batches give lower‑variance gradients but require more memory and computation. System considerations such as GPU memory often dictate the chosen batch size in practice.
Normal Equations and Linear Algebra
When the design matrix X (with rows x^{(i)T}) has full column rank, X^T X is invertible and the least‑squares solution is θ* = (X^T X)^{-1} X^T y. This solution is exact and does not require iteration. The derivation assumes that the number of examples n is at least the number of features d (plus the intercept). X^T X is positive semidefinite; invertibility requires it to be positive definite. If X^T X is singular, the solution is not unique and any vector in the null space can be added to θ* without changing the loss.
Practical Advice
Students should look at their data before modeling, as visual inspection can reveal patterns and data quality issues. The lecture encourages asking questions during live sessions and using the Friday TA sections to review notation, calculus, and linear algebra. The provided slides and course notes are recommended resources; the notes of the co‑instructor are described as the most complete reference for rigorous study.