nuglifeleoji/Sentry

Sentry – Runtime Failure Management for LLM Agents

What it is

• An external library that sits alongside a language‑model‑driven agent and watches its interaction loop. It detects when the agent makes an invalid action, gets stuck in a loop, drifts from the task, or otherwise fails at runtime.
• When a failure is spotted, Sentry proposes a repair: either a hard re‑try of the action in the correct format or a soft, guided suggestion drawn from a growing “playbook” of previously verified recoveries.
• After the repair, Sentry verifies that the agent has indeed recovered before letting the loop continue.

Why it matters

• Typical runtime monitors give static advice and can bloat the agent’s prompt. Sentry learns from each successful fix, keeping the agent’s context lean while still providing targeted help.
• The paper reports up to 37 % average performance gains on several benchmark suites (WebShop, AppWorld, SWE‑bench Lite, Mind2Web) and an 81.7 % recovery rate across 939 detected failures.

Key components

1. Failure Detection – watches recent steps for invalid actions, schema mismatches, or lack of progress.
2. Hard Repair – asks the agent to redo an invalid action in the required format.
3. Soft Repair – supplies a tailored prompt derived from a labeled “playbook” lesson that previously fixed a similar failure.
4. Recovery Verification – checks subsequent steps to confirm the issue is resolved.
5. Online Playbook Learning – verified soft repairs are stored as reusable lessons for future tasks.

Quick start (from the README)

# Clone and install
git clone https://github.com/nuglifeleoji/Sentry.git
cd Sentry
python -m venv .venv
source .venv/bin/activate
python -m pip install -e .

# Validate a config (optional)
sentry-validate configs/paper.yaml

Integrating into an agent loop

from Sentry import SentryRunner, agent_step_from_parts

runner = SentryRunner.from_yaml("configs/providers/openai_compatible.yaml")
runner.start_task(objective=task_objective, action_schema=environment_action_schema)

try:
    step = agent_step_from_parts(
        step_id=0,
        reasoning=reasoning,
        raw_action=raw_action,
        tool_name=tool_name,
        tool_args=tool_args,
        observation=observation,
        parsed_ok=True,
        schema_valid=True,
        accepted_by_environment=True,
    )
    repair = runner.step(step, terminal=False)
    if repair is not None:
        agent_messages.append(repair.prompt_text)
finally:
    runner.finalize()

Your code still generates, validates, and executes actions; Sentry only observes and intervenes when needed.

Running the AppWorld benchmark (example from README)

python3.11 -m venv .venv-appworld
source .venv-appworld/bin/activate
python -m pip install -e '.[appworld]'
appworld install
export APPWORLD_ROOT="$PWD/outputs/appworld-root"
appworld download data --root "$APPWORLD_ROOT"
export OPENROUTER_API_KEY="your-api-key"
python -m benchmarks.appworld.run --config configs/providers/openrouter.yaml

The repository includes a benchmark harness for the official AppWorld environment.

Current status

• The repo presently contains the project overview, paper, and a minimal code skeleton.
• Full implementation and documentation are slated for release in the next 2‑3 weeks, as noted in the Release section.

License – MIT (see LICENSE).

---

All information above is taken directly from the repository’s README; no additional features have been inferred.