SimplexLab/TorchJD

What it solves

TorchJD addresses the difficulty of training neural networks with multiple losses (multi-task learning), where different objectives may conflict with one another. Standard methods of simply summing losses often fail to find an optimal balance or resolve conflicts between gradients, which can hinder model performance across all tasks.

How it works

TorchJD provides two primary strategies for handling multiple losses:

1. Scalarization: This approach combines multiple losses into a single scalar value before backpropagation. It includes various methods from literature, such as geometric mean and softmax weighting.
2. Jacobian Descent: This method computes the Jacobian matrix of losses with respect to parameters and aggregates them into a single update direction. This allows the model to find conflict-free optimization directions that might be impossible to achieve with simple scalarization.

To support these, the library includes the autojac engine for computing Jacobians and the autogram engine, which computes the Gramian of the Jacobian incrementally to enable Jacobian descent on large models without storing the full matrix in memory.

Who it’s for

It is designed for researchers and developers training multi-task neural networks who need advanced optimization techniques to balance multiple competing objectives.

Highlights

• Extensive Library of Methods: Supports a wide array of scalarizers (e.g., COSMOS, FAMO, UW) and aggregators (e.g., UPGrad, MGDA, PCGrad, CAGrad).
• Memory Efficient: The autogram engine allows for Jacobian-based optimization on large models by avoiding full Jacobian storage.
• PyTorch Integration: Seamlessly integrates with PyTorch's autograd system, providing a familiar interface for computing Jacobians and aggregating gradients.
• Citations and Research-Backed: Implements state-of-the-art techniques from various academic publications.