arcee-ai/mergekit

What it solves

mergekit is a toolkit designed to combine multiple pre-trained language models into a single model without the need for additional training or the high computational cost of ensembling. It allows users to merge specialized models to create versatile ones, transfer capabilities between models, and optimize performance while keeping inference costs identical to a single model.

How it works

The tool operates directly in the weight space of models. It uses an "out-of-core" approach and lazy loading of tensors, which enables complex merges to be performed on resource-constrained hardware, including CPUs or GPUs with as little as 8 GB of VRAM. Users define their merge operations via YAML configuration files, specifying the merge method, the models to be used, and parameters like weights and densities.

Who it’s for

It is intended for AI researchers and developers who want to create new, high-performance LLMs by combining existing checkpoints without expensive retraining. It supports a wide range of architectures like Llama, Mistral, and GPT-NeoX.

Highlights

• Diverse Merge Methods: Supports a wide array of algorithms including Linear, SLERP, TIES, DARE, and Arcee Fusion.
• Memory Efficiency: Lazy loading and out-of-core processing allow for merges on low-memory hardware.
• Advanced Model Surgery: Includes "Frankenmerging" (piecewise assembly of layers) and the ability to merge dense models into a Mixture of Experts (MoE).
• Specialized Tooling: Provides utilities for LoRA extraction, tokenizer transplantation (mergekit-tokensurgeon), and multi-stage merging workflows.
• Cuda/CPU Support: Can be run entirely on CPU or accelerated via GPU.