Deep Delta Learning (DDL)

What it is

Deep Delta Learning is a research‑grade modification to the standard Transformer architecture. Instead of the usual additive residual connection, each sub‑layer reads a direction from the current residual state, compares it to a learned target, and writes back a gated rank‑1 correction along the same direction. The gate can close (identity) or open (full overwrite), giving the model a built‑in “edit” operation that can replace specific components of the residual stream while keeping the rest untouched.

Why it matters

• More expressive residuals – the update can store several value channels (d_v > 1) without widening the attention/MLP layers, so the model can keep extra information across depth at low extra compute.
• Target‑seeking bias – the update explicitly tries to drive the readout toward a learned target, which the authors show improves validation loss and one‑shot downstream accuracy on several benchmarks.
• Simple to drop‑in – the code provides drop‑in replacements for the usual GPT‑style model files; you can swap a baseline model with a DDL version by changing the import.

Key components (as described in the repo)

Component	Role
k_l (direction)	Unit vector from the attention or MLP output that selects which part of the residual to edit.
v_l (target)	Lightweight branch that produces the desired replacement value for the selected direction.
β_l (gate)	Scalar in (0,2) computed as 2·σ(...); controls how much of the old readout is erased vs. overwritten.
Expanded residual X_l ∈ ℝ^{d×d_v}	Stores multiple value channels (d_v=4 in the expanded variants) while the main transformer stays at width d.
TC / CC / EC modules	Optional tricks: TC (temporal compression) reduces the expanded state with a causal convolution, CC (channel mixing) mixes the value channels per token, and EC (initialization convolution) seeds the expanded state from token embeddings.

Results reported

Model size	Validation loss (baseline)	Validation loss (best DDL)	1‑shot accuracy (baseline)	1‑shot accuracy (best DDL)
~124 M	2.8543	2.8299 (DDL‑TC)	48.56 %	49.47 %
~353 M	2.6053	2.5758 (DDL‑CC)	53.96 %	55.14 %

The authors note that DDL variants run slightly slower and use a bit more memory (e.g., 1158 K tokens/s vs. 1509 K for the baseline at the small scale). The experiments were single‑run, not compute‑matched.

How to use the code

1. Clone the repo and install the dependencies listed in requirements.txt (Python 3.14 required).
2. Choose a model file under model/. For example, DDL-gpt-mha-rope-CC-EC.py implements the full expanded‑state version with channel mixing and initialization convolution.
3. Create a config and model exactly as you would with a Hugging Face GPT model:

   import importlib, torch
   ddl = importlib.import_module("model.DDL-gpt-mha-rope-CC-EC")
   cfg = ddl.GPTConfig()          # defaults give the ~124 M model, d_v=4
   model = ddl.GPT(cfg)
   ids = torch.randint(0, cfg.vocab_size, (1, 16))
   out = model(input_ids=ids, labels=ids)
   print(out.loss)
   

4. For GPU‑accelerated training, use the *-accelerated.py versions which rely on Triton kernels.
5. Follow the paper’s training setup (FineWeb‑Edu dataset, 49.15 B tokens) if you want to reproduce the reported numbers.

Installation

git clone https://github.com/yifanzhang-pro/deep-delta-learning.git
cd deep-delta-learning
python -m pip install -r requirements.txt
# optional: install Triton for the accelerated files
python -m pip install triton

License

The code is released under the Apache 2.0 license, allowing free use, modification, and distribution with attribution.

Who might find this useful?

• Researchers exploring alternative residual architectures for large language models.
• Engineers looking for a drop‑in way to add a low‑overhead “edit” mechanism to existing Transformer codebases.
• Anyone interested in the theoretical analysis of residual updates (spectral properties, delta‑rule interpretation).

Citations

If you use DDL in your work, cite the arXiv pre‑print:

@article{zhang2026deep,
  title   = {Deep Delta Learning},
  author  = {Zhang, Yifan and Liu, Yifeng and Wang, Mengdi and Gu, Quanquan},
  journal = {arXiv preprint arXiv:2601.00417},
  year    = {2026}
}