ServiceNow/AgentLab

What it solves

AgentLab is designed to accelerate web agent research by providing a unified framework for developing, testing, and evaluating agents that interact with web browsers. It simplifies the process of running large-scale experiments across multiple benchmarks, managing LLM APIs, and analyzing the results of agent behavior.

How it works

AgentLab integrates with BrowserGym to provide a building blocks for agent creation. It uses Ray for large-scale parallel execution of agent tasks, allowing researchers to run thousands of jobs across different seeds and configurations. The framework provides a unified LLM API supporting OpenRouter, OpenAI, Azure, and self-hosted TGI models. It also includes tools for result analysis, such as a lazy loader for experiment data and AgentXray, a Gradio-based visualizer for tracing agent actions and screenshots.

Who it’s for

It is primarily intended for AI researchers and developers building web-navigating agents, rather than as a consumer product.

Highlights

• Broad Benchmark Support: Supports a wide array of benchmarks including WebArena, WorkArena, VisualWebArena, and OSWorld.
• Parallel Execution: Leverages Ray to handle massive parallelization of agent experiments with built-in timeout management to prevent hanging jobs.
• Visual Debugging: Includes AgentXray for visual trace analysis of agent steps, actions, and screenshots.
• Reproducibility Tools: Features a dedicated ReproducibilityAgent and a reproducibility journal to track software versions and LLM changes.
• Unified API: Provides a single interface for interacting with various LLM providers.