tophant-ai/aibeat

What it solves

AI Beat provides a security evaluation framework for generative AI, focusing on identifying safety boundaries and vulnerabilities in LLMs, RAG applications, and AI agents. It moves beyond simple scoring by providing evidence-based results, distinguishing between a safe-looking final answer and a potentially unsafe execution path (e.g., an agent leaking credentials in a temporary file while providing a safe response).

How it works

The project consists of two primary tools:

• PromptBeat: A black-box evaluation engine that tests LLMs, APIs, and RAG apps by simulating attackers and judges to evaluate behavior and safety.
• AgentBeat: An extension that instrumentally evaluates agent runtimes. It collects runtime evidence, including tool calls, commands, file changes, and trace events, to determine if an agent crossed security boundaries during execution.

Both tools share a common model for scenarios, cases, and reporting. AgentBeat uses adapters to connect to agent runtimes via a language-neutral manifest and a specific HTTP protocol.

Who it’s for

It is designed for developers and security researchers who need to rigorously test the safety and security of their AI systems, particularly those deploying autonomous agents with tool-access.

Highlights

• Evidence-based evaluation: Captures traces, environment changes, and file artifacts to prove security failures.
• Symmetric workflow: Uses a shared scenario and reporting model across both black-box (PromptBeat) and runtime-instrumented (AgentBeat) testing.
• Multi-role architecture: Supports separate roles for the attacker, judge, and target model to avoid bias.
• Extensible adapter system: Allows integration with various agent runtimes through a typed SDK and manifest-based registration.