tophant-ai/aibeat

What it solves

AI Beat provides a security evaluation framework for generative AI, focusing on whether AI systems behave safely. It addresses the problem where a final answer from an AI might appear safe, but the underlying execution path (the actions the AI took to get there) may have crossed security boundaries or leaked sensitive information.

How it works

The project consists of two primary tools:

• PromptBeat: A black-box evaluation engine that tests LLMs, RAG applications, and APIs by simulating adversarial cases and using a judge model to evaluate the behavior.
• AgentBeat: An extension of PromptBeat designed for instrumented agent runtimes. It collects runtime evidence—such as tool calls, command execution, and file changes—to detect security failures that are invisible in the final response.

Both tools use a shared model for scenarios, cases, and reporting, allowing users to start with simple input-output testing and move to deep trace analysis.

Who it’s for

It is designed for developers and security researchers who need to verify the safety and security of LLMs, RAG applications, and AI agents, particularly those that have the execution power to modify files or run commands.

Highlights

• Evidence-based evaluation: Captures traces, environment changes, and file contents to prove security breaches regardless of the answer.
• Reproducible workflows: Combines scenario design, adversarial case generation, and judgment into a single pipeline.
• Adversarial testing: Supports the use of an "attacker" model to generate test cases.
• Multi-target support: Compatible with various LLM providers, HTTP agents, and coding agents like Claude Code or OpenCode.