Ollama Minions: Hybrid Local and Cloud LLM Collaboration

Researchers from Stanford Hazy Research, in collaboration with other experts, have developed a method to shift significant portions of LLM workloads from the cloud to consumer devices. By enabling small on-device models (such as Llama 3.2 running via Ollama) to collaborate with larger cloud-based models (such as GPT-4o), the framework reduces remote operational costs with minimal impact on output quality.

Hybrid LLM Protocols: Minion and MinionS

The framework introduces two distinct protocol configurations designed to optimize the balance between cost reduction and performance maintenance.

Minion Protocol

The Minion protocol allows a cloud model to chat freely with a single local model that has access to the data. This iterative communication continues until the two models reach a solution.

• Cost Efficiency: Achieves a 30.4x reduction in remote costs.
• Performance: Maintains 87% of the cloud model's original performance.

MinionS Protocol

In the MinionS protocol, the cloud model decomposes a complex task into small, bite-sized subtasks. These subtasks are then performed in parallel by small LLMs operating on chunks of the context.

• Cost Efficiency: Achieves a 5.7x reduction in remote costs.
• Performance: Maintains 97.9% of the cloud model's performance.

Technical Implementation and Setup

The Minions framework is available as an open-source project. Implementation requires a local LLM runner (Ollama) and a cloud LLM API (OpenAI).

Installation

To set up the environment, users can clone the repository and install the dependencies:

git clone https://github.com/HazyResearch/minions.git 
cd minions
pip install -e .

Local and Cloud Requirements

• Local Model: Ollama must be installed with a model such as llama3.2 pulled locally (ollama pull llama3.2).
• Cloud Model: An OpenAI API key is required to interface with models like gpt-4o.

Programmatic Usage

The minions Python package allows for the programmatic execution of both protocols.

For the Minion protocol, the Minion class is instantiated with an OllamaClient and an OpenAIClient. The process involves passing a task and context to the minion object, specifying the maximum number of communication rounds.

For the MinionS protocol, the Minions class is used. This configuration supports structured output via Pydantic models (e.g., defining an explanation, citation, and answer schema), allowing the cloud model to orchestrate parallel subtasks across the local model.