yifanzhang-pro/deep-delta-learning

What it solves

Deep Delta Learning (DDL) addresses the limitations of standard residual networks (ResNets), which rely on strictly additive updates. While additive shortcuts prevent vanishing gradients, they can limit a network's ability to model complex, non-monotonic state transitions and can suffer from feature interference as depth increases.

How it works

DDL replaces the standard additive residual connection with a Delta Operator, a learnable, data-dependent geometric transformation. Instead of simply adding a new value to the hidden state, DDL uses a rank-1 perturbation of the identity matrix.

The process involves three key components:

1. Reflection Direction (k): A normalized vector that determines the axis of transformation.
2. Scalar Gate (beta): A value between 0 and 2 that controls the transition behavior.
3. Residual Value Vector (v): The new information being injected into the state.

By adjusting the gate, the network can dynamically switch between three regimes: identity mapping (preserving signal), orthogonal projection (erasing specific information), and Householder reflection (inverting the state to model oscillatory dynamics).

Who it’s for

Researchers and AI architects designing deep neural network architectures, specifically those looking to improve how information is preserved or erased across layers in deep residual-style networks.

Highlights

• Geometric Unification: Unifies identity mapping, orthogonal projection, and geometric reflection into a single module.
• Depth-Wise Delta Rule: Applies the Delta Rule (Target - Current) over the depth dimension rather than the time dimension, allowing for selective feature rewriting.
• Spectral Control: Explicitly controls the eigenvalues of the layer-wise transition operator to maintain training stability while increasing expressivity.
• Rank-1 Update: Uses a computationally efficient rank-1 Householder update to modulate the hidden state.