dezoito/ollama-grid-search

What it solves

Ollama Grid Search automates the tedious process of manually testing different Large Language Models (LLMs), prompts, and inference parameters to find the best combination for a specific use case. Instead of one-by-one testing, it allows users to run experiments across multiple configurations simultaneously.

How it works

The tool connects to an Ollama server (local or remote) and fetches available models. Users define a set of models, prompts, and parameter values. The application then iterates through every possible combination of these inputs—essentially performing a "grid search"—and generates a set of responses for visual inspection and comparison.

Who it’s for

Developers and AI researchers who use Ollama to run local LLMs and need a systematic way to evaluate and optimize their prompts and model selection.

Highlights

• Multi-dimensional Testing: Iterate over multiple models, prompts, and parameters at once.
• A/B Testing: Directly compare the results of different prompts or models side-by-side.
• Prompt Management: Includes a built-in prompt database with autocomplete functionality (using "/").
• Performance Metrics: Optionally outputs response metadata, including inference time and tokens per second.
• Experiment Tracking: Ability to list, inspect, and download experiment logs in JSON format.
• Concurrency Control: Supports limited concurrency or synchronous calls to prevent server overload.