promptfoo/promptfoo

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

What it solves

It replaces the trial-and-error approach to developing LLM applications by providing a systematic way to evaluate prompt performance and identify security vulnerabilities before shipping to production.

How it works

Promptfoo is a CLI and library that allows developers to run automated evaluations and red-teaming exercises. It enables side-by-side comparisons of different prompts and models (such as OpenAI, Anthropic, Azure, Bedrock, and Ollama) and can be integrated into CI/CD pipelines for automated checks and code scanning for security and compliance issues.

Who it’s for

Developers building LLM-powered applications who need to ensure their AI apps are secure, reliable, and data-driven in their model selection and prompt engineering.

Highlights

  • Automated Evaluations: Test prompts and models systematically.
  • Red Teaming: Scan for vulnerabilities to secure LLM apps.
  • Model Comparison: Compare multiple LLM providers side-by-side.
  • Developer-First: Includes features like live reload, caching, and local execution for privacy.
  • CI/CD Integration: Automate checks and review pull requests via code scanning.