irthomasthomas/llm-consortium

What it solves

LLM Consortium is a plugin for the llm package that implements a model consortium system. It addresses the problem of improving the accuracy and reliability of LLM responses by orchestrating multiple diverse language models to collaboratively solve complex problems through structured dialogue, evaluation, and iterative refinement.

How it works

The system uses a parallel reasoning method where it sends a prompt to multiple models (and multiple instances of those models) in parallel. A designated arbiter model then synthesizes or ranks these responses. The process can be iterated multiple times, with the arbiter providing feedback to the models for improvement until a confidence threshold is reached or a maximum number of iterations is hit.

Key mechanisms include:

• Semantic Consensus Filtering: Uses embeddings to cluster responses and keep the densest semantic region before arbitration.
• Judging Methods: Supports different ways for the arbiter to work, including full synthesis, simple ranking, or chain-of-thought ranking.
• Geometric Confidence: Measures agreement between responses based on their position in embedding space.
• Database Logging: All interactions are stored in a SQLite database for later analysis.

Who it’s for

Developers and AI researchers who use the llm CLI tool and want to increase the quality of their LLM outputs by leveraging a consensus-based approach across different model providers.

Highlights

• Multi-Model Orchestration: Coordinate responses from multiple different models and specific instance counts per model.
• Iterative Refinement: Automatically refine outputs through multiple rounds of dialogue and evaluation.
• Advanced Arbitration: Flexible judging methods including rank-cot for structured evaluation.
• Embedding Visualization: Ability to project and export HTML visualizations of run embeddings.
• Conversation Continuation: Support for continuing previous consortium conversations using conversation IDs.
• Token Efficiency: Option to skip intermediate synthesis to focus on targeted improvement and reduce token usage.