ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model

Learned memory improves agent performance without weight updates

ALTK-Evolve enables customer service agents to achieve top-tier performance on the τ²-bench leaderboard by automatically distilling successful and failed task trajectories into a set of behavioral guidelines. These guidelines are appended to the agent's system prompt, allowing the model to improve its reliability and accuracy without requiring fine-tuning, human annotation, or modifications to the underlying model weights.

Using this approach, a basic ReAct agent powered by gpt-4.1 achieved a 67.5% Pass^4 score on the retail domain of τ²-bench as of October 12, 2026. This performance exceeds all other retail entries on the leaderboard, including systems fine-tuned on synthetic data, demonstrating that prompt-level memory can effectively bridge the performance gap between older models and frontier systems.

Methodology: Distilling guidelines from agent trajectories

ALTK-Evolve functions as a self-improving loop that analyzes an agent's training-task attempts. The system uses the same LLM powering the agent to extract specific behavioral rules—such as authentication protocols, transaction confirmation steps, and return handling procedures—from successful and failed interactions. These rules are consolidated into a markdown-formatted list and added to the agent's system prompt for subsequent tasks.

By appending these guidelines, the agent gains a "playbook" that enforces domain-specific policies and best practices. This process is entirely automated, requiring no human intervention to curate the rules, and the guidelines are applied consistently across held-out test sets to ensure generalization.

Performance and reliability gains

Guidelines improve both the capability (Mean Accuracy) and reliability (Pass^4) of agents. In the retail domain, the "lossless" configuration (incorporating all extracted guidelines) improved Pass^4 by 10 percentage points compared to the baseline. In the airline domain, while the lossless configuration improved accuracy, a more compact "sup3" core (guidelines with support ≥ 3) provided the highest reliability, indicating that the optimal volume of guidance is domain-dependent.

Crucially, these performance gains do not come at the cost of longer, more complex conversations. Analysis shows that agents using ALTK-Evolve guidelines actually complete tasks in slightly fewer turns than baseline agents, leading to lower end-to-end latency. While the increased prompt length results in higher input token costs, these costs can be largely mitigated in production environments through the use of prompt caching.

Applicability across model tiers

Learned guidelines provide significant benefits even when applied to frontier models. When testing an airline agent powered by gpt-5.5, the addition of learned guidelines improved the Pass^4 score by 15 percentage points, taking the agent to the top of the leaderboard with an 82.0% success rate. This confirms that ALTK-Evolve is not merely a patch for weaker models but a robust framework for enhancing agent behavior across different model architectures.