ALTK-Evolve Consistency Guidelines for Agent Reliability

Reducing the Consistency Gap in AI Agents

AI agents often exhibit a "consistency gap" where they can solve a task on one attempt but fail it on the next, even with identical inputs. To address this, Hugging Face and IBM Research have introduced consistency guidelines and a Consistency Analyzer within the altk-evolve toolkit, which halves the consistency gap for ReAct agents on the AppWorld benchmark, reducing it from 24.4 percentage points (pp) to 12.0pp without sacrificing average accuracy.

The Consistency Gap: Mean@k vs. Pass^k

Standard agent benchmarks typically report Mean@k (the average pass rate across k runs), which measures general capability but masks reliability. To measure actual user experience—whether an agent will succeed every time it is asked the same question—the researchers propose Pass^k: the fraction of tasks where the agent succeeds on all k runs.

Using a ReAct agent powered by GPT-4.1 on the AppWorld test_normal dataset, the researchers identified a significant reliability issue:

  • Mean@5: 77.4% (Average success rate)
  • Pass^5: 53.0% (Success on all 5 runs)
  • Consistency Gap: 24.4pp

This gap indicates that nearly a quarter of tasks are solved inconsistently, a problem that persists even at temperature 0.0 due to platform-side perturbations and the nature of LLM probability distributions.

The Mechanics of Agent "Flipping"

Agent inconsistency stems from the shape of the token probability distribution at decision points (e.g., choosing an API or an argument):

  • Sharp Distributions: Most probability mass is on one token, making the decision resilient to noise.
  • Flat Distributions: Probability mass is spread across several near-tied tokens. Small perturbations can reorder these ties, causing the agent to "flip" its decision.

Because agent trajectories chain dozens of these decisions, a small per-step chance of flipping compounds into a high probability that at least one run in a series will diverge and fail.

The Consistency Analyzer and Guideline Pipeline

To stabilize these "flip-prone" decision points, the researchers developed a two-stage diagnostic and remediation pipeline:

1. Detection via the Consistency Analyzer

The Consistency Analyzer identifies unstable steps within a single recorded trajectory. It replays each decision step by requesting k completions (default k=5) for that specific step given the existing context. This black-box approach requires no ground truth, no logits, and no end-to-end re-runs of the task, making it viable for production traffic.

2. Generation of Targeted Guidelines

Steps flagged as inconsistent are converted into consistency guidelines using the ALTK-Evolve framework. These guidelines are not task-specific trivia but generalizable rules. For example, if an agent is inconsistent in counting markers in a note, the system may generate a guideline to "use a line-anchored regex match rather than a plain substring count."

Performance Results and Generalization

Evaluations on AppWorld test_normal using GPT-4.1 showed that consistency guidelines significantly improve reliability across all difficulty tiers:

  • Aggregate Pass^5: Increased from 53.0% to 69.0% (+16.0pp).
  • Aggregate Mean@5: Increased from 77.4% to 81.0% (+3.6pp).
  • Consistency Gap: Narrowed from 24.4pp to 12.0pp.

Impact by Difficulty

  • Medium Tasks: Saw the highest absolute gain in Pass^5 (+22.9pp).
  • Hard Tasks: Saw a 45% relative increase in Pass^5 (+14.3pp).
  • Easy Tasks: Gained +12.2pp.

Generalization

Consistency guidelines generalize beyond the specific trajectory they were mined from. When applied to different but related tasks in the same scenario, Pass^5 increased by 13.0pp. Testing with a weaker model (gpt-oss-120b) showed a similar-task generalization gain of +8.7pp, suggesting the guidelines capture reusable failure patterns rather than memorizing specific traces.

Implementation Recommendations for Agent Developers

For those deploying agents in production, the researchers suggest several key practices:

  • Report Pass^k alongside Mean@k to distinguish between a reliable agent and a lucky one.
  • Monitor the hardest tasks, as the consistency gap typically widens with task difficulty.
  • Prioritize stability over model size, as consistency is orthogonal to capability; a larger model may increase average accuracy without necessarily reducing the consistency gap.
  • Utilize offline diagnosis, as the Consistency Analyzer can identify risks using only existing traces and a few targeted LLM calls without requiring live environment replays.

Sources