StonyBrookNLP/appworld

What it solves

AppWorld provides a realistic, executable testbed for evaluating AI agents that need to interact with software via APIs and code. Researchers and developers can measure how well their agents handle complex, multi-step tasks in a controlled but lifelike setting, without the cost or risk of using real services.

How it works

AppWorld simulates a world where ~100 people use 9 day-to-day apps (like Spotify or Venmo) through 457 HTTP APIs. Each task defines a supervisor, an instruction, and an initial state (databases and time). An agent must autonomously write and execute code that calls these APIs to fulfill the instruction, using helper apps like ApiDocs and Supervisor. The engine tracks actions, evaluates correctness against ground-truth setups, and records outputs for leaderboard submission. The environment is sandboxed and can run serverless using FastAPI's test client, or via Docker for isolated execution.

Who it's for

Researchers building and comparing LLM-powered agents, especially those focused on function calling, tool use, and interactive coding. It's also useful for developers testing agent frameworks or for anyone studying how well models handle realistic, multi-step digital tasks.

Highlights

• High-fidelity simulation: 9 realistic apps, 457 APIs, and ~100 simulated people with relationships.
• Natural and diverse tasks requiring rich, interactive coding and API calls.
• Comprehensive tooling: CLI for installation, data download, exploration, evaluation, and leaderboard submission.
• Includes encryption and canary strings to reduce risk of leakage into training corpora.
• Supports multiple agent frameworks and includes MCP server/client integration.
• Active leaderboard and extensive documentation, guides, and videos.