preset-io/promptimize

What it solves

Promptimize addresses the unpredictability of Large Language Models (LLMs) by bringing a test-driven development (TDD) approach to prompt engineering. It allows developers to move away from manual trial-and-error and instead use a programmatic, rigorous framework to evaluate and refine prompts at scale.

How it works

The toolkit uses a "prompt case" system where developers define specific inputs and associated evaluation functions as code. These functions score responses on a scale from 0 to 1. Promptimize organizes these cases into "Suites" that can be executed across different models, temperatures, and settings. It generates reports to track performance, identify regressions, and compare how different prompt variations perform against one another.

Who it’s for

It is designed for product builders and AI engineers who need to ensure their prompts are reliable and deterministic enough for production use in software products.

Highlights

• Configuration as Code: Prompt cases and evaluations are defined in Python, allowing for dynamic generation of test cases.
• API Efficiency: Minimizes costs by only re-assessing prompt cases that have changed.
• Weighted Evaluation: Supports assigning weights and categories to prompt cases to prioritize critical tests in reporting.
• Extensible Hooks: Provides pre_run and post_run hooks for complex validation, such as executing generated SQL code to verify the result.
• LangChain Integration: Works with LangChain under the hood but remains compatible with any Python-based prompt generation logic.