Ax-check: Measuring Agent Experience (AX) for Software Products

Ax-check is a tool designed to measure the "Agent Experience" (AX) of a software product. It evaluates whether AI agents can successfully navigate a product's onboarding process by deploying three separate agents to attempt end-to-end onboarding. The tool provides concrete guidance on necessary improvements for documentation, marketing sites, Command Line Interfaces (CLIs), Model Context Protocol (MCP) implementations, and Skills.

How Ax-check Evaluates Product Accessibility

Ax-check assesses the accessibility of a product to AI agents by simulating real-world onboarding attempts. The system assigns a grade (A through F) and a numerical score out of 100 based on the agent's ability to find information and complete setup tasks.

Key evaluation areas include:

• Documentation and Marketing Sites: The tool checks if the homepage and documentation are agent-readable. For example, sites lacking Markdown or clear setup instructions are penalized in their scores.
• CLI and MCP Integration: The tool evaluates how cleanly a product installs via CLI or MCP. Products that allow installation without requiring an account are noted as having a better agent experience.
• Pricing Clarity: The tool analyzes whether pricing information is easily discoverable and verifiable by an agent.

Analysis of AX Scoring and Results

Recent checks performed by Ax-check highlight the common friction points for AI agents:

• High Performers (Grade A): Companies like Kernel, Loops, Increase, and Snyk have achieved perfect 100/100 scores, indicating their onboarding flows are highly optimized for agentic systems.
• Moderate Performers (Grade B): Many products score in the 79-82 range. Common blockers include a lack of Markdown on the homepage or pricing and setup steps that remain unverified despite having good documentation guides.
• Low Performers (Grade F): Sites that offer no agent-readable documentation and provide only a bare homepage typically receive a score of 0/100.

Community Insights and Technical Considerations

Users and developers discussing the tool on Hacker News have raised several technical and strategic considerations regarding the optimization of AX:

• Agent-Specific Blockers: Some users reported that standard Python urllib User-Agents may be blocked by certain security configurations (e.g., 403 errors), which can inadvertently lower a product's AX score.
• Security vs. Accessibility: There is a an inherent tension between enabling fully agentic onboarding and maintaining security. One user noted that removing human identification requirements to facilitate agent onboarding could make rate limiting easier to bypass.
• Verification Challenges: Some users observed discrepancies between Ax-check's results and manual tests with LLMs like Claude, suggesting that the specific agents used by the tool may encounter different blockers than general-purpose LLM interfaces.

"Integrating live bots to demonstrate live visitor experiences is a nice touch. I got an unexpected blocker (Default Python urllib User-Agent gets 403'd by anc.dev)"

"How do you think about enabling fully agentic onboarding without proper human identification? It feels like any rate limiting would be pretty easy to get around."

Comparison with Other AX Tools

While Ax-check focuses on the end-to-end onboarding experience, other tools like anc.dev focus on a wider variety of compliance tests and the auditing of binaries to ensure CLI tooling is discoverable and useful to agentic systems.