Training a Generative Kick Drum Diffusion Model on 6GB VRAM

Training a Generative Kick Drum Diffusion Model on 6GB VRAM

Training a high-quality generative audio model does not require a massive GPU cluster or a billion-dollar budget. By utilizing a latent diffusion pipeline and aggressive compression, it is possible to train a functional generative model on consumer-grade hardware, such as a 7-year-old NVIDIA GeForce GTX 1660 SUPER with 6GB of VRAM.

The Generative Pipeline: Three Models, One Workflow

To make audio generation computationally feasible on limited VRAM, the system avoids diffusing raw audio. Instead, it uses a three-stage pipeline that compresses audio into a latent space, generates new data in that space, and then reconstructs the audio.

1. The Variational Autoencoder (VAE)

The VAE handles the compression and decompression of audio representations. It takes a log-mel spectrogram (128x173) and squashes it into a small 3D latent tensor (4x8x11). This represents a compression ratio of approximately 63x from the spectrogram and 250x from the original raw audio.

To ensure the latent space is continuous and usable for generation, the VAE employs two mechanisms:

  • Mean and Variance Output: The encoder outputs both a mean and variance, forcing the decoder to learn to reconstruct from a neighborhood of points rather than a single coordinate.
  • KL Divergence Penalty: A penalty is applied to pull the latent distributions toward a standard normal distribution, preventing the model from creating "scattered islands" of data.

2. The Diffusion U-Net

The Diffusion U-Net is the core generative engine. It operates exclusively within the 4x8x11 latent space. The model is trained to reverse a process where Gaussian noise is sequentially added to a latent tensor over 1,000 steps.

To guide the generation, the model uses text conditioning (keywords extracted from filenames) and classifier-free guidance (CFG). By training with 15% keyword dropout, the model learns both generic and specific kick drum characteristics, allowing the user to amplify the influence of specific keywords (e.g., "punchy" or "warm") during inference.

3. The Vocoder (HiFi-GAN)

Because converting audio to a mel spectrogram is a lossy, non-invertible process, a vocoder is required to turn the generated spectrogram back into a waveform. A Generative Adversarial Network (GAN) is used here, consisting of a generator that creates audio and a discriminator that attempts to distinguish it from real training data. This adversarial process ensures the final audio sounds "real" by filling in the fine details lost during the spectrogram conversion.

Data Preparation and Preprocessing

The model was trained on a dataset of 13,615 kick samples extracted from a personal library. The following preprocessing steps were critical for model stability:

  • Filtering: Files containing "loop" or "BPM" were removed, and files smaller than 5KB or larger than 1MB were discarded.
  • Normalization: All samples were resampled to 44.1kHz, padded or trimmed to exactly 2 seconds, normalized to -1dB peak, and given a 0.2-second fade-out.
  • Spectrogram Conversion: Audio was converted to log-mel spectrograms using a window of 2048 samples and a hop length of 512 samples, resulting in a 128x173 representation.

Deployment and Inference Strategy

To avoid the cost of a 24/7 GPU instance, the project uses Modal, a serverless GPU platform.

  • Class-Based Workers: To mitigate "cold boot" times, the inference engine is wrapped in a class. This allows the models (approximately 300MB of weights) to be loaded into GPU memory once during container boot and reused for subsequent requests.
  • Performance: Warm requests take approximately 2.5 seconds, while cold boots take closer to 10 seconds.
  • Caching: Model weights are hosted on HuggingFace and cached on the Modal machine to avoid repeated downloads.

Technical Trade-offs and Artifacts

Training on limited hardware necessitates certain compromises. The author noted a "granularization" artifact—where the kick sounds as if it is composed of tiny grains rather than a continuous wave—especially when heavy compression (via OTT) is applied. This is attributed to the high compression ratio of the latent space (63x) and the upsampling process of the vocoder.

Future Improvements for Model Quality

To reduce these artifacts and improve conditioning, the following hyperparameters were identified for tuning:

  • Latent Size: Increasing the latent tensor size to reduce compression loss.
  • FFT Window: Reducing the 2048-sample window to sharpen transients.
  • Keyword Dropout: Tuning the 15% dropout rate to improve the responsiveness to text prompts.
  • Vocabulary: Retaining numeric identifiers (like "808" or "909") which carry significant sonic meaning in electronic music.

Summary of Key Takeaways

Component Strategy Benefit
Hardware GTX 1660 SUPER (6GB VRAM) Proves consumer hardware is sufficient for specialized generative tasks.
Representation Log-Mel Spectrogram $\rightarrow$ Latent Reduces computational load by diffusing in a compressed space.
Architecture VAE $\rightarrow$ U-Net $\rightarrow$ HiFi-GAN Separates compression, generation, and reconstruction.
Inference Serverless GPU (Modal) Eliminates idle GPU costs while maintaining acceptable latency.

Sources