Stanford CS329A Self-Improving AI Agents Part 7: Self-Improvement and Deep Research Agents

TL;DR

Improving AI agents via search works in two ways: (1) generating many code samples, filtering and clustering them to pick diverse candidates, which scales solve rate with sample budget but is limited by selection bottlenecks; (2) letting a large reasoning model trigger search queries when it expresses uncertainty, then reasoning over retrieved documents (Search‑O1), which outperforms standard and agentic retrieval‑augmented generation on GPQA and multi‑hop question answering. AlphaCode2 demonstrates that a stronger base model (Gemini Pro) plus a learned scoring model reduces the required sample budget from 1 million to ~100 samples while raising solve rate from 25 % to 43 % and reaching an 85th‑percentile ranking in competitive programming.

AlphaCode: Sampling, Filtering, and Clustering

AlphaCode solves competitive‑programming problems by pretraining a masked language model on GitHub and CodeContests data, then generating 1 million diverse candidate programs per question (half Python, half C++). It filters candidates that pass the given example tests, clusters them to keep syntactically different but semantically equivalent solutions, and submits a curated subset to the Codeforces platform. On 10 contests with ~5 000 participants each, AlphaCode achieved an average ranking of 54.3 % (assuming 10 submissions per problem) and was competitive with 28 % of competitors from the last six months. The solve rate scales roughly log‑linearly with the number of samples, but when only 10 submissions are allowed the selection and clustering stage becomes a bottleneck, limiting accuracy to ~30 % versus >40 % with unlimited attempts.

AlphaCode2: Fine‑tuning Gemini Pro and Learned Scoring

AlphaCode2 replaces the pretrained model with a fine‑tuned Gemini Pro, uses a family of fine‑tuned variants to increase diversity, and introduces a learned scoring model (reward model) that predicts correctness of code samples. The data mix includes the open‑source CodeContests V2 set and a higher‑quality curated set for training the scorer. With the same 1 million‑sample budget, AlphaCode2 reaches a 43 % solve rate, nearly double AlphaCode’s 25 %. More importantly, AlphaCode2 matches AlphaCode’s solve rate with only ~100 samples per problem, showing that a stronger base model and better scoring reduce the required sampling budget. In percentile terms, AlphaCode2 scores around the 85th percentile among expert and master‑candidate human contestants on Codeforces, whereas AlphaCode was around the 46th percentile (or 99.5 % when considering top‑two solutions).

Search‑O1: Triggering Search When the Model Expresses Uncertainty

Search‑O1 builds deep‑research agents on top of large reasoning models. Instead of a single retrieval step, the model generates a search query whenever its reasoning chain contains uncertainty indicators (e.g., “perhaps”, “alternatively”, “wait”). The retrieved document is then analyzed to extract only the relevant chunks, which are inserted back into the prompt, allowing the model to continue coherent reasoning. This iterate‑retrieve‑reason loop reduces uncertainty propagation and avoids overwhelming the model with irrelevant text. On GPQA (physics, chemistry, biology) and multi‑hop QA benchmarks such as HotpotQA, 2Wiki, MusiQue, and Bamboogle, Search‑O1 outperforms both standard retrieval‑augmented generation and agentic RAG, achieving accuracy that is competitive with human experts in physics and chemistry and approaching expert levels in biology.

Search‑R1 vs. Search‑O1: Prompting vs. Reinforcement Learning

The lecture contrasts Search‑O1’s prompting‑based approach (model learns when to search via explicit uncertainty tokens) with Search‑R1’s reinforcement‑learning‑based approach, which teaches the model to decide when to issue search queries through an RL loop. Search‑R1 is not covered in detail but is noted as a complementary direction for automatically learning search behavior.

Key Takeaways

  • Scaling sample generation improves solve rate, but selection and clustering become bottlenecks when submission budget is limited.
  • A stronger base model (Gemini Pro) combined with a learned scoring model yields large gains: AlphaCode2 reaches AlphaCode’s performance with ~1 % of the sample budget and nearly doubles absolute solve rate.
  • In reasoning‑heavy tasks, triggering search queries when the model signals uncertainty and reasoning over retrieved documents (Search‑O1) is more effective than naïve retrieval‑augmented generation, especially for multi‑hop questions.
  • The methods demonstrated in AlphaCode/AlphaCode2 and Search‑O1 illustrate two complementary paths to self‑improving agents: (1) better search over output space via sampling and learned scoring, and (2) better interaction with external knowledge via uncertainty‑driven search.

Sources