arcee-ai/mergekit

Tools for merging pretrained large language models.

What it solves

mergekit provides a way to combine multiple pre-trained language models into a single model without the need for additional training or the high computational cost of ensembling. This allows users to combine specialized capabilities, transfer skills between models, and find optimal performance trade-offs while keeping the inference cost identical to that of a single model.

How it works

The toolkit operates directly in the weight space of models. It uses an "out-of-core" approach and lazy loading of tensors, which allows it to perform complex merges on resource-constrained hardware, running entirely on CPU or with as little as 8 GB of VRAM. Users define their merge operations via a YAML configuration file, specifying the merge method, the models or layers (slices) to be used, and the parameters for the merge.

Who it’s for

AI researchers and developers who want to create new, high-performance language models by merging existing ones without needing massive datasets or compute resources for training.

Highlights

  • Low Resource Requirements: Supports GPU or CPU execution with lazy loading to minimize memory usage.
  • Wide Model Support: Compatible with Llama, Mistral, GPT-NeoX, StableLM, and others.
  • Diverse Merge Methods: Includes a variety of techniques such as Linear, SLERP, TIES, DARE, and Mixture of Experts merging.
  • Advanced Capabilities: Supports "Frankenmerging" (piecewise assembly of layers), LoRA extraction, and evolutionary merge methods.
  • Flexible Tokenizer Handling: Provides fine-grained control over vocabulary alignment and embedding transplantation.

Related

  • Project
    Tencent/AngelSlim

    AngelSlim is a comprehensive toolkit for compressing and accelerating large AI models, including LLMs, vision, and speech models, through techniques like quantization and speculative decoding.

  • Project
    ModelCloud/GPTQModel

    GPT‑QModel is an open‑source Python library that unifies many low‑bit quantisation methods (GPTQ, AWQ, etc.) for LLMs, offering calibration, conversion, validation, and fast inference across CUDA, ROCm, Ascend, Intel XPU, Apple Silicon and CPU, with integrations for Transformers, vLLM and SGLang.

  • Project
    texttron/tevatron

    Tevatron is a unified toolkit for training and evaluating neural retrievers and rerankers. It supports large language models on GPUs and TPUs, offers LoRA-based efficient tuning, and includes ready-to-use datasets and evaluation workflows for text, image, and multilingual retrieval.

  • Project
    open-compass/VLMEvalKit

    An open-source evaluation toolkit for large vision-language models that enables one-command evaluation across 70+ benchmarks and 200+ models.

  • Project
    feature-engine/feature_engine

    A Python library providing Scikit-learn compatible transformers for comprehensive feature engineering and selection to improve machine learning model performance.