ServiceNow/AgentLab

What it solves

AgentLab provides a standardized framework for developing and evaluating web-based AI agents. It simplifies the process of running large-scale experiments across multiple benchmarks, managing the complexity of browser interactions, and analyzing the results of agent performance in dynamic web environments.

How it works

AgentLab integrates with BrowserGym to provide a set of building blocks for agent creation. It uses a unified LLM API to connect to various providers (OpenRouter, OpenAI, Azure, or self-hosted TGI) and leverages Ray for large-scale parallel execution of agent tasks. The framework includes a Study system to manage experiments, a SequentialStudies object to handle task dependencies in specific benchmarks, and a dedicated visualization tool called AgentXray for inspecting agent traces.

Who it’s for

It is designed for researchers and developers working on web agent research who need to run extensive benchmarks, conduct ablation studies, and ensure the reproducibility of their results.

Highlights

• Broad Benchmark Support: Compatible with a wide array of benchmarks including WebArena, WorkArena, VisualWebArena, AssistantBench, and OSWorld.
• Large-Scale Parallelism: Uses Ray to run dozens of jobs in parallel on a single machine, with built-in timeout handling to prevent hanging jobs.
• Unified LLM Interface: A single API for switching between different LLM backends easily.
• AgentXray Visualization: A Gradio-based interface for visualizing agent traces, actions, and screenshots to debug and analyze performance.
• Reproducibility Tools: Includes a reproducibility journal and a specialized ReproducibilityAgent to compare execution diffs across different runs.