ianarawjo/ChainForge

What it solves

ChainForge addresses the difficulty of systematically testing and comparing LLM prompts. Instead of relying on ad-hoc chatting with a single model, it provides a structured environment to battle-test prompt variations, different models, and various model settings to determine which combination yields the highest quality responses.

How it works

It uses a visual data-flow environment (built with ReactFlow and Flask) where users can create chains of nodes. The system employs "combinatorial power," taking the cross product of input variables to automatically generate and run every possible permutation of a prompt across multiple selected LLMs. Users can then apply Python-based evaluation nodes to score responses and visualization nodes to plot the results (e.g., using box-and-whisker plots or histograms).

Who it’s for

It is designed for prompt engineers and AI researchers who need to move beyond manual testing to a more rigorous, scalable approach for evaluating LLM behavior and optimizing prompt templates.

Highlights

• Multi-Model Querying: Send the same prompt permutations to multiple LLMs simultaneously.
• Combinatorial Testing: Automatically test all combinations of prompt templates and input variables.
• Integrated RAG Support: Includes browser-based and server-side nodes for uploading, chunking, retrieval, and reranking.
• Visual Evaluation: Built-in tools to visualize numeric and boolean metrics from LLM responses.
• GenAI Assistance: Features to create synthetic data tables and generate starter code for evaluation functions.