Stanford CS329A Self-Improving AI Agents – Course Overview (Fall 2025)

Scaling Laws and Emergent Capabilities

The lecture shows that increasing model parameters, training compute, and dataset size consistently lowers test loss, which underlies the performance gains from GPT‑2 through GPT‑4.

  • As model size grows from BERT (~340 M) to GPT‑2 (1.5 B), GPT‑3 (175 B), PaLM (540 B) and estimated trillion‑parameter GPT‑4, test loss decreases, yielding better language models.
  • Larger models exhibit few‑shot and zero‑shot learning: a model can follow a task description or a few examples without explicit fine‑tuning.
  • Emergent reasoning appears only in larger models; chain‑of‑thought prompting improves performance on math and reasoning tasks when the model is sufficiently big (e.g., LaMDA, GPT, PaLM at ≥8 B parameters).
  • These scaling trends motivated further work on instruction tuning and reinforcement learning from human feedback to turn raw language models into usable assistants.

Instruction Tuning and Reinforcement Learning from Human Feedback

Instruction tuning and RLHF were the key steps that transformed base models into systems like ChatGPT.

  • After pre‑training on next‑token prediction, models are fine‑tuned on high‑quality supervised data (books, essays) to improve general language ability.
  • Instruction tuning presents question‑answer pairs (optionally with chain‑of‑thought) so the model learns to follow directions and produce answers.
  • RLHF builds a reward model from human rankings of model outputs; the model is then optimized to maximize this reward, aligning outputs with human preferences such as correctness, helpfulness, or harmlessness.
  • The combination of pre‑training → high‑quality fine‑tuning → instruction tuning → RLHF produced the capabilities seen in ChatGPT, surpassing earlier models like GPT‑3.

Inference‑Time Scaling (Large Language Monkeys)

Inference‑time scaling can unlock additional capability without changing model parameters.

  • The Large Language Monkeys project repeatedly samples a model’s output for a given problem and uses a verifier to select correct answers, analogous to the infinite monkey theorem.
  • Experiments increased samples per problem from 1 to 10,000 on math and coding benchmarks; models that were weaker than GPT‑4o with a single sample surpassed it when many samples were allowed.
  • This demonstrates that models already know more than what a single greedy decode reveals; inference scaling (e.g., repeated sampling, tree search) can improve pass‑@k or coverage metrics.
  • The approach is sample‑efficient: a modest number of parallel samples can yield correct answers, and latency can be mitigated by parallel generation.
  • When verifiers are unavailable, researchers explore LLM‑as‑judge or learned reward functions to provide feedback.
  • Combining inference‑time scaling with synthetic data generation enables self‑improvement loops: models produce candidate solutions, verifiers filter them, and the filtered data fine‑tunes the model further.

From LLMs to Agentic Workflows

The course moves beyond single‑turn chatbots to agents that accomplish multi‑step goals using tool use, planning, and self‑correction.

  • Agents differ from chatbots by maintaining a goal, planning actions, interacting with external tools (e.g., web search, code execution), and using feedback to adjust.
  • Workflow patterns include prompt chaining (subtask sequencing), routing (simple vs. complex paths), parallelization (simultaneous LLM calls on different inputs), and orchestrator‑worker (a planner LLM dispatches worker LLMs).
  • Verifiers (unit tests, rule‑based checkers) and critics (LLM‑as‑judge) provide feedback for self‑correction and backtracking.
  • Examples discussed: Claude Code for coding assistance and deep‑research tools for end‑to‑end literature synthesis, both of which rely on the above patterns.
  • Effective agents require strong planning, multi‑step reasoning, and the ability to revise steps based on tool output or verifier signals.

Course Logistics

The course structure, assessments, and expectations are outlined as follows.

  • Prerequisites: comfort with machine learning basics and deep learning (as listed on the course website).
  • Materials: lecture slides, readings, and videos are posted on Canvas and the public site cs329a.stanford.edu.
  • Assessment: three homeworks (50 % of grade) and a course project (50 % of grade).
  • Homework schedule and due dates are available on Canvas; late days are permitted per the posted policy.
  • The project can be done individually or in teams of up to four; API credits are provided.
  • Milestones: project proposal (early October), midterm presentation (two weeks after proposal), final report and poster session (December 12, 4‑6 PM).
  • Projects should be research‑oriented (hypothesis‑driven, not merely an app); past projects have led to conference publications.
  • Office hours and Q&A are handled via Ed (public forum) and Gradescope for submission.
  • Lecture videos will eventually be posted on YouTube for audit‑only viewing (no credit).

Sources