stevibe/BenchLocal

What it solves

BenchLocal provides a local-first desktop environment to test and compare Large Language Models (LLMs) on real-world tasks. It eliminates the friction of manually testing models by allowing users to run standardized "Bench Packs" (benchmark sets) against multiple local or remote models side-by-side.

How it works

The software acts as a shared desktop runtime that manages provider configurations, model registries, and the execution of benchmarks. Users install Bench Packs—which contain scenario definitions, prompts, and scoring logic—and run them through the app. The system supports per-tab sampling overrides and maintains a history of run results.

Additionally, it offers an "Agent Access" feature via an HTTP API and Model Context Protocol (MCP), allowing AI agents and automation tools to control the benchmark workflows programmatically while the UI remains active.

Who it’s for

It is designed for developers and AI researchers who need to evaluate LLM performance on specific tasks (such as tool calling, bug finding, or data extraction) using a consistent framework across different models.

Highlights

• Local-First Desktop App: A dedicated environment for managing and executing LLM benchmarks.
• Bench Pack System: Support for installable, modular benchmark sets with their own scoring and verifier logic.
• Agentic Control: Integration with MCP and OpenAPI, enabling external AI agents to manage runs and models.
• Side-by-Side Comparison: Ability to compare multiple models' outputs on the same tasks simultaneously.