magenta/ddsp

What it solves

DDSP provides a way to combine traditional digital signal processing (DSP) with deep learning. It solves the problem of creating audio generation models that are interpretable and physically meaningful, rather than treating audio as a raw waveform or a black-box output of a neural network.

How it works

The library provides differentiable versions of common DSP functions—such as synthesizers, filters, and waveshapers—that can be used as layers within a neural network. Instead of the network predicting raw audio samples, it predicts the controls (like pitch and amplitude) for these DSP processors.

Key components include:

• Processors: Modules that convert neural network outputs into physically meaningful controls to generate or transform audio signals.
• ProcessorGroup: A system to chain multiple processors together in a Directed Acyclic Graph (DAG), which can be configured via .gin files for flexible experimentation without changing Python code.
• Core Library: A collection of differentiable DSP functions, spectral operations, and loss functions specifically designed for audio applications.

Who it’s for

It is designed for researchers and developers working on audio synthesis, timbre transfer (e.g., converting a voice to a violin), and self-supervised pitch detection.

Highlights

• Interpretable Audio Generation: Uses physically based synthesis instead of raw waveform generation.
• Timbre Transfer: Includes pretrained models to convert audio between different sound sources.
• Flexible Configuration: Supports complex signal processing chains via ProcessorGroup and gin dependency injection.
• Differentiable Components: All DSP functions are differentiable, allowing them to be trained using standard gradient-based optimization.