Armature: Product Analytics and Evals for MCP Agent Sessions
Armature is a product analytics platform designed specifically for agent sessions occurring within AI clients like Claude, ChatGPT, and Cursor. Unlike traditional UI analytics, Armature captures interactions that happen inside the AI client's interface, providing visibility into how users interact with products via Model Context Protocol (MCP) servers, Claude Connectors, and ChatGPT Apps.
Automated Use Case Discovery and Ranking
Armature uses models to analyze every session to identify the user's primary intent. This allows product teams to group sessions into specific use cases and rank them by volume and success rate. Crucially, the platform identifies "unsupported" use cases—requests users are attempting to make that the product does not yet support—providing a direct roadmap for feature development based on actual user demand.
Issue Identification and Root Cause Analysis
Armature scans sessions for failures, loops, and dead ends to group them by root cause. The platform can identify failures even when the API responses return a 200 OK status, meaning it can detect logical failures where the agent fails to achieve the user's goal despite the technical success of the API calls. Examples of identified issues include:
- Agent Loops: Repeated attempts to call a function due to missing authentication scopes.
- Search Misses: Failures to occur because of specific phrasing (e.g., "refund").
- Pagination Errors: Data truncation caused by pagination limits.
- Rate Limiting: Failures during bulk update operations.
Session Replay and Evaluation
Every session is scored based on whether the user's intent was successfully fulfilled. Developers can replay the full trace of a session, which includes the initial user request, the agent's internal thinking process, and every subsequent tool call. This visibility allows teams to pinpoint exactly where a session broke down.
Integration and Deployment
Armature is integrated via an SDK that wraps an existing MCP server, Claude Connector, or ChatGPT App backend. According to the company, the SDK can be implemented in a few minutes by generating an API key and using a coding agent to wire the SDK into the deployment.
Supported Clients
Armature supports any client capable of reaching an MCP server, including:
- Claude
- ChatGPT
- uma Cursor
- Codex
- Gemini CLI
Data Privacy and Security
To handle sensitive information, Armature employs detection models that scan sessions and redact personally identifiable information (PII) and secrets by default before the data reaches storage. Users can control data retention policies and delete data at any time.
Comparison to Existing Tooling
Armature distinguishes itself from other observability and analytics tools based on the target user and the type of data captured:
| Tool Category | Focus | Armature's Difference |
|---|---|---|
| UI Analytics (e.g., PostHog, Amplitude) | Human clicks in a proprietary UI. | Captures sessions happening inside third-party AI clients where the product owner has no UI control. |
| Agent Observability (e.g., LangSmith, Langfuse) | Engineering-focused observation of agents built by the developer. | Product-focused analysis of how users' agents experience the product. |
Community Feedback and Questions
One community member raised questions regarding the technical implementation and the necessity of Armature over existing logging frameworks:
"How is it actually getting the model's thoughts? ... If im already logging MCP calls with OpenTelementry or BrainFuse why would i need Armature?"
These questions highlight a tension between raw technical logging (via OpenTelemetry) and the high-level product analytics (intent discovery and success scoring) provided by Armature.