catid/dora
Implementation of DoRA
Overview
This project provides a specific implementation of DoRA (Weight-Decomposed Low-Rank Adaptation), a technique for fine-tuning large neural networks efficiently. The code is based on an original implementation but includes one key change: the trainable magnitude values are stored in FP32 (a high-precision floating-point format) instead of lower precision. This change is intended to improve training stability or accuracy.
The repository includes:
- A custom PyTorch layer (
DoRALayer) that applies DoRA to a linear layer. - A helper function (
replace_linear_with_dora) to swap standard linear layers in a model with DoRA layers. - A test suite verifying correctness.
- An extensive experimental study comparing 76 different initialization methods for the low-rank factors, with detailed results on several tasks.
How DoRA Works
For a given weight matrix W (shape [out_features, in_features]), DoRA computes an adapted weight as:
V = W + A @ B
W_adapted = m * (V / column_norm(V))
AandBare low-rank matrices (the “adapter”) that are learned during fine-tuning.mis a per-column magnitude vector (shape[1, in_features]) that scales each column ofVafter normalizing it.- The normalization divides each column by its Euclidean norm, so the magnitude is controlled separately by
m.
The original implementation initializes A with random normal values and B with zeros, so the initial adapted weight equals the original W. The only modification in this project is that m is stored in FP32 (instead of the default dtype) to preserve precision during training.
Usage
To use the code, install dependencies and run the main script or tests:
python -m pip install -r requirements.txt
python dora.py
python -m unittest -v test_dora
In Python, you can create a DoRA layer from an existing linear layer:
import torch
from torch import nn
from dora import DoRALayer, replace_linear_with_dora
base = nn.Linear(128, 64)
layer = DoRALayer(
d_in=128, d_out=64, rank=8,
weight=base.weight.detach().clone(),
bias=base.bias.detach().clone(),
)
# The DoRA layer should produce the same output as the original linear layer initially.
You can also replace all linear layers in a model with DoRA layers (rank 4 by default):
model = nn.Sequential(nn.Linear(128, 64), nn.ReLU(), nn.Linear(64, 16))
replace_linear_with_dora(model)
Experimental Results
The README reports a large, audited study comparing different initialization strategies for the low-rank factors (A and B). The study tested 76 families of initializers on tasks like SciFact, NFCorpus, ViGGO, and Aircraft. Key findings:
- Some initializations improve performance on specific tasks (e.g., IniLoRA-alpha on SciFact, Student-t on NFCorpus, Truncated normal on ViGGO, LoRA-GA on Aircraft).
- Improvements are not universal – some tasks show no significant gain, and some initializers perform worse.
- The study includes fresh-seed confirmations and detailed statistical intervals, but the results are task-specific and not a general ranking.
For example, on SciFact, the best initialization (IniLoRA-alpha) improved the nDCG@10 score from 67.99 to 70.37 (on fresh seeds), but the confidence interval includes zero, meaning the improvement is not statistically certain. On Aircraft, the gain was +6.62 points with a clear positive interval.
The README also documents a separate warm-start substudy and an extension with LoRA-Null and GME recipes, where most variants did not beat the accepted baseline.
Summary
This project is a focused implementation of DoRA with a precision tweak, plus a comprehensive empirical evaluation of initialization methods. It provides ready-to-use code and evidence that initialization choices can matter, but the best method depends on the task. The repository is well-documented with reproducible experiments and audits.
Related
- Project
- Dispatch
- Dispatch
- Dispatch
- Project