Gauge CLI: Open-Source Tool for Improving Agent Experience
Overview
Gauge is an open-source command-line interface (CLI) designed to improve "Agent Experience" (AX) by enabling developers to test how effectively AI coding agents interact with their products. By running real coding agents against local, containerized environments, Gauge helps identify where agents struggle, such as when they encounter blocking interactive commands, rely on deprecated APIs, or fail to discover available tools.
The Improvement Loop
Gauge facilitates a "hillclimbing" process—a continuous cycle of measurement, analysis, and refinement. The workflow operates as follows:
- Define Tasks: Developers create evaluation cases using Markdown or SQLite-backed prompts that define specific criteria for success.
- Test in Sandbox: Using Docker, Gauge spins up isolated environments where agents attempt to complete tasks using the developer's current code, documentation, or skills.
- Identify Friction: Gauge normalizes transcripts from various agents (including Claude Code, Codex, Pi, and OpenCode) to pinpoint where the agent became blocked or confused.
- Refine and Retest: Developers apply fixes to their documentation, help text, or code, then rerun the evaluation to verify that the friction has been resolved.
Local vs. Hosted Execution
Gauge offers two modes of operation to accommodate different development needs:
- Local Mode: Runs entirely on the developer's machine using Docker. It requires no Gauge account and keeps all data, including SQLite measurements and transcripts, local to the project. This is ideal for rapid iteration on working-tree changes.
- Hosted Gauge: Provides managed infrastructure for teams. It enables shared results, automated judging, and the ability to test more complex inputs like authenticated APIs or MCPs (Model Context Protocol) servers that may require persistent sessions.
Key Capabilities
Gauge is built to integrate directly into the developer's existing workflow. Key features include:
- Skill Testing: Validates whether agents can successfully discover and utilize custom "Skills" or instructions provided in the repository.
- Evidence Collection: Automatically captures structured findings and judgments, allowing developers to see exactly why an agent failed a specific criterion.
- Comparison Tools: Provides commands to compare batches of test results, making it easy to see if a change improved performance or introduced new issues.
- Web Fixtures: Allows developers to test changes to documentation or web-based content before they are officially published, ensuring that the agent's environment reflects the intended future state.
Getting Started
To begin using Gauge, developers install the CLI via npm and pull the required runtime:
npm install -g @withgauge/cli
gauge local runtime pull
After installation, gauge init --local creates a private workspace within the project directory. From there, developers can define evaluation cases and install skills to begin testing their agent-facing interfaces. The tool is licensed under the MIT license and is available on GitHub.