Stanford CS329A Lecture 4: Learning from Feedback with Tools/Code – ReAct, RLEF, and Constitutional AI

ReAct: Combining Reasoning and Tool Use

ReAct shows that prompting a model to interleave verbal reasoning traces with tool calls yields better grounding and more interpretable behavior on knowledge‑intensive tasks. By first generating a reasoning trace, then selecting an action based on that trace, observing the result, and repeating the loop, the model can retrieve external information (e.g., via web search) and incorporate it into its reasoning. This approach reduces hallucination compared to pure chain‑of‑thought because the model can verify its answers against an external knowledge base. On HotpotQA, FEVER, and WebShop, ReAct outperforms action‑only baselines and, when combined with chain‑of‑thought self‑consistency or a fallback mechanism, exceeds standard techniques. In the WebShop decision task, ReAct achieved a score of 66.6 while human experts scored 82.1, indicating remaining headroom but demonstrating the value of grounding reasoning in tool use.

RLEF: Learning Code from Execution Feedback

RLEF (Reinforcement Learning from Execution Feedback) improves code generation by treating test execution as the feedback signal in a PPO training loop. The model receives a natural language problem, generates code, runs it against a public set of tests for immediate inference‑time feedback, and iterates until the code passes or a turn limit is reached. The final solutions are then evaluated on a hidden private test set, whose outcome provides the reward for the PPO update. This two‑tier test strategy prevents the model from simply memorizing public test outputs. A hybrid token‑turn level policy computes value functions at the turn level while generating code token by token, giving fine‑grained control over generation. Experiments on CodeContests show higher solve rates (e.g., solve rate 10 at k improves with training) and fewer wrong outputs as the model learns to repair its code using execution feedback. The binary reward from test pass/fail is sufficient for these relatively short competitive‑programming problems, and the method generalizes to other code‑generation benchmarks.

Constitutional AI: Self‑Critique via Human‑Written Principles

Constitutional AI trains a model to be harmless by using a human‑written set of principles (the Constitution) as a source of AI‑generated feedback. The process has two stages: first, the model critiques its own outputs against the Constitution and revises them; these self‑critique/revision traces are used for supervised fine‑tuning. Second, a preference model is trained on the revised outputs and the Constitution, and this preference model guides reinforcement learning to produce responses that are both helpful and harmless. Results show that the harmlessness Elo score increases substantially, while helpfulness may slightly decrease; combining Constitutional AI with chain‑of‑thought yields the best Pareto frontier between helpfulness and harmlessness. The approach scales because only a small amount of post‑training compute is needed to update the Constitution, and it avoids the need for large‑scale human labeling required by standard RLHF.

Discussion: Challenges and Extensions

The lecture highlighted several open challenges. For ReAct, noisy or misleading environment feedback can be mitigated by adding a reflection layer or backtracking mechanism, and confidence can be improved through repeated sampling and majority voting. Extending ReAct to larger action spaces requires more demonstrations and increases inference cost, suggesting a need for efficient action selection. In RLEF, binary feedback may be insufficient for complex debugging tasks; richer error traces could improve repair efficiency. For Constitutional AI, updating the Constitution without full retraining remains an open continual‑learning question, and ensuring the model does not over‑rely on self‑critique (given potential overconfidence) may benefit from ensemble‑based critique. The discussion also noted that humans employ additional cognitive mechanisms—task decomposition, parallel thinking, memory, and adaptive reasoning‑to‑action ratios—that could be incorporated into future agent designs.