ALTK-Evolve: Calibrating Agentic Memory for LLM Performance

IBM Research has demonstrated that agentic memory—the process of distilling lessons from past trajectories into guidelines—is not a universal feature but a "dose" that must be calibrated to a model's specific capabilities. Using the ALTK-Evolve framework, researchers found that while strong models benefit from comprehensive guideline sets, weaker models perform better with a compact core and task-specific retrieval, and some saturated models show no gain at all.

The ALTK-Evolve Memory Loop

ALTK-Evolve implements a learning loop that happens around the model rather than inside it, meaning no model weights are updated and no human annotation is required. The process follows four primary steps:

1. Trajectory Generation: The agent attempts tasks and produces execution trajectories.
2. Guideline Extraction: ALTK-Evolve extracts behavioral guidelines (strategies that worked, mistakes to avoid, and edge cases) from both successful and unsuccessful runs.
3. Consolidation: These guidelines are consolidated into a reusable set.
4. Inference Injection: At inference time, the agent is provided with either the full guideline set or a curated selection of these guidelines.

Model Capability and Memory Dosage

Evaluation across eight models spanning various sizes and architectures reveals three distinct patterns regarding how models absorb agentic memory:

Strong Models with Headroom

Models with significant capacity can absorb and apply a full set of guidelines, including those for rare edge cases. For example, DeepSeek-V3.2 (671B MoE) saw a +9.5 percentage point (pp) increase in Task Goal Completion (TGC) when provided with its full self-mined guideline set.

Weak or Selective Models

Smaller or weaker models can be overwhelmed by large guideline sets. These models perform best with a "curated retrieval" strategy: a high-confidence core of guidelines supplemented by a few task-relevant ones. gpt-oss-120b (117B MoE) achieved a +16.1pp gain in TGC using curated retrieval, whereas the full guideline set was less effective and increased token costs by approximately 50%.

Saturated Models

Some models show no measurable gain from agentic memory, regardless of the configuration. GLM-5 (745B MoE) exhibited this pattern, suggesting the model may have already reached its performance ceiling on the tested tasks or cannot effectively apply the provided guidance.

Performance Benchmarks on AppWorld

The framework was evaluated on AppWorld, consisting of 585 multi-step tasks across nine simulated applications. Performance was measured using Task Goal Completion (TGC) and the stricter Scenario Goal Completion (SGC), which requires an agent to pass every variant of a scenario to succeed.

Model	Pattern	Baseline TGC / SGC	Best-memory TGC / SGC	Best config	$\Delta$ TGC	$\Delta$ SGC
gpt-oss-120b	Weak / selective	39.9 / 21.4	56.0 / 37.5	curated retrieval	+16.1	+16.1
DeepSeek-V3.2	Strong w/ headroom	79.8 / 64.3	89.3 / 80.4	full guideline set	+9.5	+16.1
Claude Opus 4.6	Strong w/ headroom	90.5 / 87.5	94.6 / 94.6	full guideline set	+4.1	+7.1
GPT-5.5	Strong (near-ceiling)	92.3 / 82.1	95.2 / 89.3	full guideline set	+2.9	+7.2
GLM-5	Saturated	87.5 / 80.4	87.5 / 80.4	full guideline set	0.0	0.0

Notably, SGC gains were often larger than TGC gains, indicating that self-distilled guidelines significantly improve agent reliability across scenario variants.

Cost and Efficiency Trade-offs

Injecting memory increases the input token count for every ReAct step, but the impact varies by strategy:

• Curated Retrieval Efficiency: For models like gpt-oss-120b, curated retrieval provided the highest accuracy gain (+16.1pp TGC) with minimal overhead (+5% tokens).
• Full Set Overhead: DeepSeek-V3.2 experienced a +78% increase in tokens per task when using the full guideline set.
• Optimization via Caching: Because the guideline set is static across steps, prompt caching can substantially reduce the effective cost of the "full guideline set" strategy in production.

Future Research Directions

IBM Research identifies several areas for further development of the ALTK-Evolve framework:

• Learned Selectors: Replacing cosine similarity retrieval with a selector trained on outcome signals to better predict which guidelines help specific tasks.
• Teacher-Distilled Memory: Exploring memory for very weak models that lack the internal signal necessary for self-distillation.
• Context Window Isolation: Conducting controlled experiments to determine if context-window size, rather than raw capability, influences a model's ability to absorb full guideline sets.