Stanford CS229 Spring 2026 Lecture 7: Neural Networks 1 (Architecture)

Introduction

The lecture serves as an introduction to deep learning, focusing on neural networks as a structured nonlinear model for supervised learning tasks such as regression and classification.

Framework for Supervised Learning

The supervised learning framework remains the same for nonlinear models: define a loss function for each example, average over the dataset, and minimize the resulting objective with respect to the model parameters.

Loss Functions

For regression, the loss is the squared error (y_i − h_θ(x_i))², which can be derived from a Gaussian likelihood. For multiclass classification, the model outputs logits via h_θ̄(x) and the loss is the cross‑entropy loss −log(exp(h_θ̄(x)_y) / Σ_j exp(h_θ̄(x)_j)).

Optimization via Gradient Descent

To minimize the loss, gradient descent updates θ ← θ − η ∇J(θ). Computing the full gradient is expensive, so stochastic gradient descent (SGD) samples a single example or a mini‑batch to obtain an unbiased estimate of the gradient. The noise in SGD can help escape sharp minima in non‑convex landscapes, though the theory is limited.

Neural Network Building Blocks

A neuron applies an affine transformation followed by a nonlinear activation. The rectified linear unit (ReLU) is defined as ReLU(t) = max(t, 0) and is widely used because it preserves linearity for positive inputs and zeroes out negative ones. Other activations such as sigmoid, tanh, and leaky ReLU are mentioned as alternatives.

Multilayer Networks and Notation

Stacking affine‑activation pairs yields a multilayer perceptron (MLP). In matrix form, a layer computes a = ReLU(Wx + b); subsequent layers repeat the operation with their own weight matrices and bias vectors. The overall number of layers equals the number of activation applications. Parameters are the collection of all W and b matrices.

Residual Networks

Residual (skip) connections add the layer’s input to its output: Z ← σ(W₂σ(W₁Z + b₁) + b₂) + Z. This formulation encourages the network to learn residual updates, eases optimization, and was popularized by the 2015 ResNet paper.

Layer Normalization

LayerNorm normalizes a feature vector z by subtracting its mean and dividing by its standard deviation, then scales and shifts with learnable parameters γ and β: y = γ ⊙ (z − μ)/σ + β. Variants such as RMSNorm omit the mean subtraction. Normalization stabilizes training by making the layer’s output insensitive to the scale of its inputs.

Convolutional Networks (Brief)

Convolutional layers replace dense matrix multiplication with a sparse, structured matrix that shares weights across spatial locations, enabling efficient processing of grid‑structured data such as images. The lecture notes cover this topic, though it is used less frequently in modern vision models that favor transformers.