Stanford CS229 Spring 2026 Lecture 14: Transformers and In‑Context Learning
TL;DR
The lecture explains how large language models turn text into tokens, model token sequences autoregressively with a transformer, train by maximizing likelihood, and generate text using techniques like temperature scaling and top‑k sampling, while noting that self‑attention scales quadratically with sequence length.
Tokenization
Tokenization converts raw text into a sequence of integer IDs that the transformer can process. The speaker describes tokenization as choosing the smallest input unit, noting that pure character or word tokenization is inefficient. Subword tokenization (e.g., byte‑pair encoding) splits words into reusable pieces, allowing the model to share understanding across related words such as "internationalize" and "internationalization). The resulting vocabulary is a predefined list of subwords; each token receives an ID. The speaker mentions that open‑source models such as Qin 3.5 have roughly 250 tokens in their vocabulary (as per the speaker’s remark) and that some proprietary tokenizers, like those used by Cloud Code, have been made more granular, increasing the token count for the same text from about 1,000 to 1,500 tokens.
Autoregressive Probability Model
The language model defines a probability distribution over token sequences using the chain rule: p(x₁…x_T) = ∏ₜ p(x_t | x₁…x_{t‑1}). Each conditional distribution is modeled by a neural network that outputs a logit vector over the vocabulary, which is turned into a probability via softmax. Because directly modeling the joint distribution would require V^T} is intractable, the autoregressive decomposition reduces the per step to a softmax over V‑is manageable.
Transformer Architecture
The transformer computes a sequence of token t depends only on preceding tokens, the model is causal (auto‑regressive). During training, the loss is the negative log‑likelihood of the observed sequence, computed as the sum of −log softmax(f_θ(x₀…x_{t‑1}))[x_t]. For generation, the model samples token by token from these conditional distributions; temperature scaling sharpens or softens the distribution, and top‑k sampling limits consideration to the k most likely tokens.
Attention Mechanism
A single‑head attention layer takes a sequence of vectors (the token embeddings) and produces a new sequence of vectors. For each position t, it computes queries Q_t = h_t W_Q, keys K_t = h_t W_K, and values V_t = h_t W_V (all row vectors). Attention scores are the inner products Q_t·K_sᵀ for all source positions s, scaled and passed through a softmax to obtain weights α_{t,s}. The output at position t is the weighted sum ∑s α{t,s} V_s. To enforce causality, a mask sets future‑position scores to −∞ before the softmax, ensuring that α_{t,s}=0 for s>t.
Multi‑Head Attention and Masking
Multiple attention heads operate in parallel, each with its own projection matrices (W_Q, W_K, W_V). Their outputs are concatenated and linearly projected to produce the final layer output. The speaker notes that the number of heads is typically on the order of tens to hundreds, depending on model size. Masking is applied inside each head to preserve the auto‑regressive property: before softmax, any score corresponding to a future token is replaced by −∞, which becomes zero after softmax, thus preventing the model from attending to future tokens.
MLP and Residual Connections
After each attention layer, each position’s vector passes through a multi‑layer perceptron (MLP) consisting of two linear layers with a non‑linearity (e.g., GeLU). The MLP is applied independently and identically at every position. Residual connections add the layer’s input to its output, followed by layer normalization (the speaker mentions pre‑norm or post‑norm variants). This pattern—attention → add & norm → MLP → add & norm—is repeated across multiple transformer layers.
Training and Generation
Training minimizes the negative log‑likelihood summed over all time steps, using optimizers such as SGD or Adam (referred to as a black‑box optimizer). Generation starts from a beginning‑of‑sentence token (or a given prompt) and iteratively samples the next token from the softmax distribution. Temperature t scales the logits before softmax: t<1 sharpens the distribution (favoring high‑probability tokens), t>1 softens it (increasing diversity). Top‑k sampling keeps only the k largest logits, renormalizes, and samples from that subset.
Computational Efficiency
The speaker highlights that naive self‑attention requires computing all pairwise inner products, leading to O(T²·d_h) time and memory, where T is sequence length and d_h is the head dimension. This quadratic dependence becomes prohibitive for long contexts (e.g., millions of tokens). Techniques such as flash attention aim to reduce memory footprint by recomputing parts of the attention matrix on‑the‑fly. The speaker notes that alternative attention variants can improve the dependence on T but may trade off expressiveness.
Based solely on the provided transcript; no external facts were added.