catid/dora

Implementation of DoRA

Overview

This project provides a specific implementation of DoRA (Weight-Decomposed Low-Rank Adaptation), a technique for fine-tuning large neural networks efficiently. The code is based on an original implementation but includes one key change: the trainable magnitude values are stored in FP32 (a high-precision floating-point format) instead of lower precision. This change is intended to improve training stability or accuracy.

The repository includes:

  • A custom PyTorch layer (DoRALayer) that applies DoRA to a linear layer.
  • A helper function (replace_linear_with_dora) to swap standard linear layers in a model with DoRA layers.
  • A test suite verifying correctness.
  • An extensive experimental study comparing 76 different initialization methods for the low-rank factors, with detailed results on several tasks.

How DoRA Works

For a given weight matrix W (shape [out_features, in_features]), DoRA computes an adapted weight as:

V = W + A @ B
W_adapted = m * (V / column_norm(V))
  • A and B are low-rank matrices (the “adapter”) that are learned during fine-tuning.
  • m is a per-column magnitude vector (shape [1, in_features]) that scales each column of V after normalizing it.
  • The normalization divides each column by its Euclidean norm, so the magnitude is controlled separately by m.

The original implementation initializes A with random normal values and B with zeros, so the initial adapted weight equals the original W. The only modification in this project is that m is stored in FP32 (instead of the default dtype) to preserve precision during training.

Usage

To use the code, install dependencies and run the main script or tests:

python -m pip install -r requirements.txt
python dora.py
python -m unittest -v test_dora

In Python, you can create a DoRA layer from an existing linear layer:

import torch
from torch import nn
from dora import DoRALayer, replace_linear_with_dora

base = nn.Linear(128, 64)
layer = DoRALayer(
    d_in=128, d_out=64, rank=8,
    weight=base.weight.detach().clone(),
    bias=base.bias.detach().clone(),
)
# The DoRA layer should produce the same output as the original linear layer initially.

You can also replace all linear layers in a model with DoRA layers (rank 4 by default):

model = nn.Sequential(nn.Linear(128, 64), nn.ReLU(), nn.Linear(64, 16))
replace_linear_with_dora(model)

Experimental Results

The README reports a large, audited study comparing different initialization strategies for the low-rank factors (A and B). The study tested 76 families of initializers on tasks like SciFact, NFCorpus, ViGGO, and Aircraft. Key findings:

  • Some initializations improve performance on specific tasks (e.g., IniLoRA-alpha on SciFact, Student-t on NFCorpus, Truncated normal on ViGGO, LoRA-GA on Aircraft).
  • Improvements are not universal – some tasks show no significant gain, and some initializers perform worse.
  • The study includes fresh-seed confirmations and detailed statistical intervals, but the results are task-specific and not a general ranking.

For example, on SciFact, the best initialization (IniLoRA-alpha) improved the nDCG@10 score from 67.99 to 70.37 (on fresh seeds), but the confidence interval includes zero, meaning the improvement is not statistically certain. On Aircraft, the gain was +6.62 points with a clear positive interval.

The README also documents a separate warm-start substudy and an extension with LoRA-Null and GME recipes, where most variants did not beat the accepted baseline.

Summary

This project is a focused implementation of DoRA with a precision tweak, plus a comprehensive empirical evaluation of initialization methods. It provides ready-to-use code and evidence that initialization choices can matter, but the best method depends on the task. The repository is well-documented with reproducible experiments and audits.

Related

  • Project
  • Dispatch
  • Dispatch
  • Dispatch
  • Project