teticio/audio-diffusion

What it solves

This project enables the synthesis of music and audio loops by applying diffusion models—techniques typically used for image generation—to audio data. It provides a framework to train and use models that can generate new audio samples based on learned patterns from a dataset of audio files.

How it works

Audio is first converted into mel spectrograms (visual representations of sound), which are treated as images. The project uses several diffusion techniques to generate these spectrograms:

• DDPM (De-noising Diffusion Probabilistic Models): Trains on mel spectrograms to synthesize similar ones, which are then converted back into audio.
• DDIM (De-noising Diffusion Implicit Models): Allows for faster generation with fewer steps and enables deterministic encoding for interpolation between audio samples in noise space.
• Latent Audio Diffusion: Uses a Variational AutoEncoder (VAE) to compress audio into a lower-dimensional latent space, making training and inference faster and more efficient.
• Conditional Generation: Allows the model to generate audio based on specific encodings (such as text or audio features) to influence the output.

Who it’s for

It is designed for musicians, audio engineers, and AI researchers interested in generative audio, music synthesis, and creating audio variations or remixes.

Highlights

• Flexible Generation: Supports generating audio from scratch, creating variations of an existing track, or "out-painting" by masking parts of the input audio.
• Slerp Interpolation: Ability to interpolate between two different audio samples in latent space.
• Pre-trained Models: Includes various pre-trained models for different genres, such as Instrumental Hip Hop and Spotify-liked playlists.
• Integration: Migrated to the Hugging Face diffusers package for easier use and contribution.