Hugging Face PEFT New LoRA Merging Methods

Hugging Face has integrated several new merging methods into the PEFT library, allowing developers to combine multiple LoRA adapters derived from the same base model. This update enables the on-the-fly synthesis of model capabilities, allowing a single merged adapter to perform tasks that require the combined knowledge of multiple specialized adapters.

Supported LoRA Merging Methods

PEFT now supports a variety of merging algorithms, each handling the adapter weights (matrices $A$ and $B$) differently to balance performance and memory usage.

Concatenation (cat)

Concatenation is the exact weighted merging of LoRA adapters. It concatenates the $A$ and $B$ matrices of participating adapters, allowing them to have different ranks. While this method typically yields great results, combining a large number of adapters can lead to increased adapter size and potential Out-of-Memory (OOM) errors.

Linear/Task Arithmetic (linear)

Based on the "Editing Models with Task Arithmetic" paper, this method performs a weighted sum of the individual $A$ and $B$ matrices. This approach requires all participating LoRA adapters to have the same rank.

Singular Value Decomposition (svd)

SVD considers the product $BA$ (the delta weight) as the task weight. It computes the merged delta weight and then applies SVD to approximate the new $A$ and $B$ matrices. This method supports adapters with different ranks and allows users to specify the rank of the resulting merged adapter, though it is GPU memory-intensive.

TIES (ties, ties_svd)

TIES (TRIM, ELECT SIGN & MERGE) resolves interference when merging models. It prunes the smallest values of task weights (based on a density fraction), calculates a majority sign mask (using either total magnitude or frequency of signs), and performs a disjoint merge. It is available in both standard and SVD variants.

DARE (dare_linear, dare_ties, dare_linear_svd, dare_ties_svd)

DARE randomly prunes task weights based on a 1-density fraction and rescales the remaining weights by 1/density. DARE acts as a plug-in that can be combined with Linear/Task Arithmetic or TIES methods.

Magnitude Prune (magnitude_prune, magnitude_prune_svd)

This method prunes the smallest values of the task weights based on a density fraction and then performs a weighted sum of the remaining task tensors. It is available in both standard and SVD variants.

Implementation and Capabilities

Users can implement these methods using the add_weighted_adapter() class method in PEFT.

Combining Specialized Capabilities

Merging adapters allows a model to support use cases that neither individual adapter could handle alone. For example, combining a "mental health" adapter with a "Hinglish" (Hindi-English) language adapter enables the model to provide concrete mental health suggestions in Hinglish, whereas the individual adapters could only provide the suggestions in English or the language style in Hinglish without the specialized knowledge.

Extension to Text-to-Image Generation

These merging methods are applicable to text-to-image generation via the 🤗Diffusers library. By obtaining PeftModels from LoRA checkpoints and using add_weighted_adapter(), users can combine different style or subject LoRAs (e.g., combining a "toy-face" LoRA with a "Pixel-Art" LoRA) to generate images that blend both characteristics.

Technical Observations and Recommendations

Based on Hugging Face's testing, the following guidelines are recommended for selecting a merging method:

  • Starting Point: Start with the cat method for the best results in most scenarios, provided the number of adapters is small.
  • Alternative Linear Methods: If cat is insufficient, try linear, magnitude_prune, and dare_linear in that order. For magnitude_prune and dare_linear, a density value between 0.7 and 0.8 is recommended.
  • TIES Configuration: When using ties, majority_sign_method="frequency" often performs better than the default total. A recommended starting density is 0.5.
  • SVD Variants: Use the *svd family of methods when working with Stable Diffusion LoRA adapters of different ranks. Note that SVD operations are computationally expensive and require more GPU memory.
  • DARE Performance: The dare_ties variant was observed to not produce good results.

Sources