BrowserGym – A Framework for Web‑Agent Research

What it is

BrowserGym is an open‑source Python library that turns a web browser (via Playwright) into a Gymnasium environment. Researchers can plug in any LLM‑based or rule‑based agent and let it interact with real‑world web pages or synthetic web tasks. The library ships with a collection of well‑known web‑automation benchmarks (MiniWoB, WebArena, WorkArena, etc.) and makes it easy to add new tasks by subclassing a single AbstractBrowserTask class.

Why it matters

Web‑agent research—training LLMs to browse, fill forms, click buttons, or complete multi‑step workflows—needs a standardised, reproducible playground. BrowserGym provides:

• A common API (gym.make(...)) so the same agent code works across many benchmarks.
• Plug‑and‑play benchmarks covering synthetic, visual, and real‑world tasks.
• Integration with the companion AgentLab platform for large‑scale evaluation and trace collection.
• Extensibility for custom tasks, enabling rapid prototyping of new web‑interaction challenges.

Who it’s for

• Researchers building or evaluating LLM‑driven web agents.
• ML engineers who need a reproducible environment for benchmarking new prompting or fine‑tuning strategies.
• Developers creating custom web‑automation tasks who want to reuse the Gym‑style loop and built‑in metrics.

Getting started (quick‑start)

# Install the full package (includes all benchmarks)
pip install browsergym
# Install Playwright’s Chromium engine
playwright install chromium

Then launch a simple open‑ended task:

import gymnasium as gym
import browsergym.core  # registers the environment

env = gym.make(
    "browsergym/openended",
    task_kwargs={"start_url": "https://www.google.com/"},
    wait_for_user_message=True,
)
obs, info = env.reset()
while True:
    action = ...          # your agent’s next action
    obs, reward, done, _, info = env.step(action)
    if done:
        break
env.close()

Replace the ... with calls to an LLM (e.g., OpenAI’s gpt‑4o‑mini) or any custom policy.

Main features (from the README)

Feature	Details
Benchmarks bundled	MiniWoB, WebArena, WebArenaVerified, VisualWebArena, WorkArena, AssistantBench, WebLINX, OpenApps, TimeWarp, Knows
Extensible task class	Inherit AbstractBrowserTask to create new benchmarks
Modular installation	Separate pip packages (browsergym‑core, browsergym‑miniwob, …) let you install only what you need
Playwright backend	Uses Chromium via Playwright for reliable, headless browsing
Gymnasium API	gym.make, env.step, env.reset – familiar to RL/agent developers
AgentLab integration	Companion repo for large‑scale experiment management and trace analysis
Demo agent	Ready‑to‑run script (demo_agent/run_demo.py) that works with any OpenAI model
Open‑source & MIT‑licensed	Free to use, modify, and extend

Ecosystem & related projects

• AgentLab – framework for running agents on BrowserGym benchmarks and analysing results.
• WorkArena – ServiceNow‑specific benchmark for enterprise‑style tasks.
• WebArena / VisualWebArena – realistic web‑task suites on self‑hosted domains.
• MiniWoB – classic synthetic web‑task collection.
• DoomArena – security‑testing arena that can inject attacks into BrowserGym pages.

When to use it vs. alternatives

Situation	BrowserGym is a good fit
You need a standardised RL‑style interface for web tasks	✅
You want to evaluate agents on multiple public benchmarks without rewriting wrappers	✅
You only need a single, simple web‑scraping script	❌ (lighter tools like Selenium or Playwright alone are enough)
You require visual perception (screenshots) as part of the observation	✅ (VisualWebArena and screenshot options are built‑in)

Citation

If you use BrowserGym in a paper, cite the two provided bibtex entries (2025 Transactions on Machine Learning Research and 2024 ICML). Traces from the accompanying research are hosted on Hugging Face.

---

BrowserGym is a research framework, not a consumer product. Use responsibly.