ChainForge: a visual toolkit for prompt engineering and LLM hypothesis testing

What it solves

ChainForge provides a visual environment for "battle-testing" prompts and comparing LLM responses. It moves beyond ad-hoc chatting to allow users to systematically analyze how different prompt variations, model settings, and different LLMs themselves affect the quality of the responses generated.

How it works

It uses a data-flow programming model (built on ReactFlow and Flask) where users create chains of nodes. Key capabilities include:

• Combinatorial Prompting: It takes the cross-product of input variables, allowing users to send hundreds of queries across multiple prompt templates and model permutations simultaneously.
• Multi-Model Querying: Users can query multiple LLM providers (including OpenAI, Anthropic, Google Gemini, DeepSeek, and local models via Ollama) at once.
• Evaluation and Visualization: It includes evaluation nodes for scoring responses (via Python scripts) and visualization nodes to plot numeric or boolean metrics (e.g., box-and-whisker plots or histograms).
• GenAI Assistance: Built-in features help create synthetic data tables and generate starter code for evaluation functions.

Who it’s for

Prompt engineers, AI researchers, and developers who need to robustly verify model behavior and find the optimal prompt and model combination for a specific use case.

Highlights

• Visual Interface: A node-based environment for designing prompt chains and evaluation flows.
• Broad Provider Support: Compatible with a wide range of cloud APIs and locally-hosted models.
• Exportable Data: Ability to export results to spreadsheets (Excel .xlsx) for further analysis.
• Ground Truth Evaluation: Support for importing datasets to compare LLM responses against expected answers.