ifixai-ai/iFixAi

What it solves

Existing AI evaluation tools often focus on technical metrics like latency or token efficiency. iFixAi addresses the gap by auditing AI agents based on business KPIs and organizational structure to uncover misalignment, mistakes, and blind spots before they cause operational failures.

How it works

iFixAi treats the AI agent as a black box and runs a series of diagnostic probes. It employs a dual-role system to ensure objectivity: a System Under Test (SUT) is the agent being graded, and an independent Judge (from a different AI vendor) evaluates the results. This prevents the agent from grading itself and provides a citable grade.

Users can run the tool via a guided CLI wizard, explicit CLI flags for automation, or as a plugin/skill integrated directly into AI agents like Claude Code, Cursor, or VS Code. It supports a wide range of providers (OpenAI, Anthropic, Gemini, etc.) and can test either bare model APIs or real deployed agents via HTTP endpoints.

Who it’s for

Developers and organizations deploying AI agents who need operational assurance and independent auditing to ensure their agents are behaving according to business rules and safety policies.

Highlights

• Independent Auditing: Uses a cross-vendor judge system to provide objective, citable grades.
• Multi-modal Interface: Available as a CLI, a native plugin for Claude Code/Codex, or a scaffolded skill for various IDE-based agents.
• ** uma Comprehensive Scoring**: Grades agents on a weighted average of five core pillars: Fabrication, Manipulation, Deception, Unpredictability, and Opacity.
• Flexible Testing: Supports testing of bare models or fully deployed agents via OpenAI-compatible HTTP endpoints.
• Tiered Test Suites: Offers various suites (smoke, strategic, core, extended, all) to balance speed and depth of audit.