ALTK-Evolve vs ACE: Same Lessons, Fewer Tokens

TL;DR

ALTK‑Evolve achieves equal or better task‑goal completion than ACE while consuming roughly one‑quarter to one‑seventh of the tokens, because it delivers only the most relevant guidelines at inference time instead of always feeding the entire playbook.

---

What Both Systems Agree On

Both ACE (Agentic Context Engineering) and ALTK‑Evolve treat an agent’s past trajectories as lessons that are stored without compression. Each lesson is kept as a distinct object with a support count indicating how many episodes produced it. This design avoids the brevity bias and context collapse problems that arise when a model is forced to summarize a large body of experience into a few generic rules.

"Count them, don't collapse them." – the shared principle behind ACE’s per‑bullet counters and ALTK‑Evolve’s support counts.

Where the Systems Differ

Memory Construction

• ACE builds a single evolving playbook via a Generator → Reflector → Curator loop. Near‑duplicate lessons are clustered and merged; the surviving entry inherits the combined support count, preserving experience provenance.
• ALTK‑Evolve creates a flat set of guidelines, each with its own support count. Lessons are not merged into a single document; instead they remain individually retrievable.

Delivery Mechanism (Token Cost Driver)

• ACE injects the entire playbook into the prompt at every reasoning step, regardless of model size or task difficulty.
• ALTK‑Evolve treats delivery as a configurable dial: a small core of high‑support guidelines is always included, and additional guidelines are selected per‑task using cosine similarity or LLM‑guided ranking. When a model has sufficient context window, the full consolidated set can be used; otherwise only the most relevant subset is sent.

The delivery choice directly explains the token‑usage gap reported below.

---

Empirical Comparison on the AppWorld Benchmark

All experiments use the same ReAct code‑agent base and identical task splits (AppWorld test_normal, 168 tasks). Results are single‑run pass@1 scores.

Model	System	TGC ↑	SGC ↑	Tokens per task
DeepSeek‑V3.2	ACE	80.4 / 73.2	634 K
	ALTK‑Evolve	89.3 / 80.4	263 K
gpt‑oss‑120b	ACE	54.8 / 35.7	777 K
	ALTK‑Evolve	56.0 / 37.5	116 K

On the stronger model (gpt‑oss‑120b), ALTK‑Evolve attains a higher TGC while using only ~15% of ACE’s token budget. On the weaker model (DeepSeek‑V3.2) it matches ACE’s accuracy at ~40% of the token cost.

Difficulty‑Based Breakdown

DeepSeek‑V3.2 (right panel of Figure 1)

• ALTK‑Evolve wins on Easy, Hard, and Overall.
• ACE slightly edges on Medium.

gpt‑oss‑120b (left panel of Figure 1)

• ACE leads on Easy and Medium because a full playbook helps a strong model with generic instructions.
• ALTK‑Evolve dominates Hard tasks, where selecting the right lesson matters more than sheer volume, and this advantage lifts the aggregate score.

The figure illustrates that selective retrieval benefits harder tasks, while a full playbook can be advantageous for easier, more generic subtasks.

---

Why Delivery Matters

The token disparity originates entirely from the delivery stage. ACE’s fixed‑size injection forces the model to process the entire playbook each step, inflating the prompt length regardless of the model’s context capacity. ALTK‑Evolve’s adaptive retrieval trims the prompt to what the model can actually attend to, reducing inference cost without sacrificing (and sometimes improving) performance.

Try the ALTK‑Evolve library for the extraction, consolidation, and retrieval pipeline, or read the full technical report for detailed ablations.

---

Linked Artifacts & References

• ALTK‑Evolve introduction – https://huggingface.co/blog/ibm-research/altk-evolve
• ACE (Agentic Context Engineering) – https://arxiv.org/abs/2510.04618
• AppWorld benchmark – https://appworld.dev/appworld
• ALTK‑Evolve code – https://github.com/AgentToolkit/altk-evolve
• Full technical report – https://arxiv.org/abs/2603.10600

---

Methodological Details

• Benchmark: AppWorld test_normal, 168 multi‑step tasks across nine simulated apps.
• Agent: ReAct code agent (Python‑generating steps, environment returns outputs).
• Metrics: TGC (Task Goal Completion) and SGC (Scenario Goal Completion). Tokens counted per task, including all prompt material.
• Memory source: Lessons mined from training/dev splits only; no human labels.
• Baseline differences: The ReAct prompt template differs between the two systems, leading to slightly different no‑memory baselines (72.0 vs 79.8 TGC). Comparisons focus on relative gains after adding memory, not on the absolute baseline gap.

---

Reference Tables

DeepSeek‑V3.2 – test_normal

System	# Guidelines	TGC	SGC	Tokens/task
ReAct, no memory	0	79.8	64.3	148 K
ReAct + ACE	106	80.4	73.2	634 K
ReAct + ALTK‑Evolve	191	89.3	80.4	263 K

gpt‑oss‑120b – test_normal

System	# Guidelines	TGC	SGC	Tokens/task
ReAct, no memory	0	39.9	21.4	110 K
ReAct + ACE (full)	full	54.8	35.7	777 K
ReAct + ALTK‑Evolve (selected)	~29	56.0	37.5	116 K

gpt‑oss‑120b – Difficulty Breakdown (TGC)

Difficulty	Baseline	ACE	ALTK‑Evolve
Easy	66.7	84.2	82.5
Medium	35.4	60.4	56.2
Hard	19.1	23.8	31.8
Aggregate	39.9	54.8	56.0

---

Takeaway

Both ACE and ALTK‑Evolve prove that agentic memory—turning an agent’s own experience into reusable lessons—improves multi‑step task performance without any weight updates. The decisive factor for practical deployment is how those lessons are delivered: a calibrated, task‑aware retrieval strategy (ALTK‑Evolve) can retain or improve accuracy while slashing inference token consumption dramatically.