AgentEnv Framework – Quick Overview

What it is – A Python SDK and command‑line tool (agent‑env) for creating, deploying, and evaluating agentic environments (containerised services that talk the agentenv‑framework‑protocol). It lets you package an environment as a Docker image, run it behind a gateway, point an LLM‑style agent at it, and automatically score the agent’s behaviour.

Key pieces

• Environments – Containerised servers (local Docker, Vercel sandbox, or custom micro‑VMs) that expose a standard protocol.
• Agents – Your own LLM‑oriented code (or external services) that interact with those environments via the protocol.
• Tasks / Episodes – Defined scenarios that orchestrate turns between agents and environments, handling time‑outs, tool calls and scoring.
• Back‑ends – Local Docker, Vercel sandbox, smol_vm micro‑VMs, plus optional cloud stores for artifacts (AWS, GCP, TensorLake, Tenki).
• Plugins – Extensible points for new env types, providers, task steps, or storage back‑ends.

Typical workflow

1. Define an environment (Dockerfile + agentenv‑framework‑protocol server).
2. Build & register it with agent‑env (creates a versioned image).
3. Create a task that describes the episode (turn order, prompts, scoring).
4. Run the task via agent‑env run <task> or from Python using the SDK:

   spec = EpisodeSpec.from_task(Task.get("my-task"), turn_timeout=900)
   episode = await spec.start()
   while (turn := await episode.next_turn()) is not None:
       async with turn.tools() as tools:
           ...  # call tools, read prompts, produce response
       await turn.end(response="Done.")
   scores = await episode.result()
   

5. Collect scores – The same numbers shown by the CLI are returned for reinforcement‑learning or evaluation.

Installation – Available on PyPI as agentenv-framework. Requires Python ≥ 3.11.4 and Docker for local back‑ends. Install the CLI with uv tool install agentenv-framework (or pip) and add extras for specific providers ([vercel], [smol], [aws], etc.).

Why it matters – It abstracts away the plumbing of spinning up isolated services and scoring LLM agents, making it easier to benchmark or train agents that need to call external tools, APIs, or custom code.

Resources

• Docs: https://www.agentenvframework.com/docs
• GitHub: https://github.com/scaleapi/agentenv-framework
• License: Apache‑2.0