SREGym/SREGym

Can AI agents resolve production incidents?

What it solves

SREGym provides a high-fidelity benchmarking platform for AI agents designed for Site Reliability Engineering (SRE). It addresses the lack of live, realistic environments where AI agents can be tested on their ability to diagnose and mitigate real-world cloud failures, such as CPU exhaustion, TLS mismatches, and network policy blocks.

How it works

The platform creates live system environments (using Kubernetes clusters, either self-managed or emulated via Kind) where agents are deployed in isolated Docker containers. These agents attempt to solve a suite of 90 different SRE problems, moving through two primary stages: diagnosis and mitigation. The system uses a judge model to evaluate the agent's performance and supports a wide range of LLM providers via LiteLLM, including local models via Ollama.

Who it’s for

It is designed for researchers and developers building next-generation AI agents for AIOps and SRE, as well as those looking to evaluate the effectiveness of LLM-based troubleshooting tools in complex cloud environments.

Highlights

  • Extensive Problem Set: Includes 90 SRE problems, covering OS-level faults, metastable failures, and concurrent failures.
  • Real-world Simulations: Simulates high-profile outages like the Cloudflare WAF regex CPU exhaustion and the Reddit Pi-Day outage.
  • Flexible Infrastructure: Supports both full Kubernetes clusters for leaderboard-grade results and a "svelte" profile for memory-constrained local iteration.
  • Isolated Execution: Agents run in hardened Docker containers with filtered network access to ensure security and and prevent access to internal grading logic.

Related

  • Project
    ServiceNow/BrowserGym

    BrowserGym is an open‑source Python library that turns web browsing into Gymnasium environments, letting researchers plug LLM‑based agents into many web‑automation benchmarks (MiniWoB, WebArena, WorkArena, etc.). It provides a unified API, modular install options, Playwright‑driven browsers, and easy extensibility via an `AbstractBrowserTask` base class. The ecosystem includes AgentLab for large‑scale evaluation and a demo agent that works with OpenAI models.

  • Project
    Purewhiter/mobilegym

    A browser-hosted mobile simulation platform for GUI agent research that provides programmable state and deterministic judges for scalable RL training and verifiable evaluation.

  • Dispatch
    Engrim 1.3.0: Local‑First SQLite Memory Engine for Multi‑Model AI CLIs

    Engrim 1.3.0 introduces a universal, project‑scoped SQLite memory store that lets developers switch between AI coding assistants without losing context, cutting reload costs by over 99%.

  • Project
    nudgebee/nudgebee

    Nudgebee is an open-source SRE copilot that automates observability, incident response, and FinOps across Kubernetes and cloud environments using LLM-driven triage and runbook automation.

  • Project
    IBM/AssetOpsBench

    AssetOpsBench is an open‑source benchmark/framework for building, orchestrating, and evaluating LLM‑based AI agents that operate on industrial asset‑management data (sensors, work orders, failure modes, etc.). It provides domain‑specific MCP tool servers, several ReAct‑style agent back‑ends, 141+ realistic scenarios, and a multi‑dimensional evaluation pipeline used in KDD/AAAI/NeurIPS competitions.