AudioLDM 2 Optimization Guide: Reducing Inference Time with Hugging Face Diffusers

Hugging Face has released an optimization guide for AudioLDM 2, a text-to-audio latent diffusion model capable of generating realistic sound effects, human speech, and music. By applying a combination of code and model optimizations within the diffusers library, inference time for a 10-second audio sample can be reduced from over 30 seconds in the original implementation to less than 1 second.

AudioLDM 2 Model Architecture

AudioLDM 2 is a text-to-audio latent diffusion model (LDM) that generates audio by learning continuous representations from text embeddings. The generation process follows a multi-stage pipeline:

1. Text Encoding: The model uses two encoders—the text-branch of CLAP (for audio-aligned embeddings) and the text-encoder of Flan-T5 (for semantic representation)—to compute text embeddings.
2. Projection: These embeddings are passed through linear projections via the AudioLDM2ProjectionModel into a shared embedding space.
3. Auto-regressive Generation: A GPT2 language model generates a sequence of $N$ embedding vectors conditional on the projected CLAP and Flan-T5 embeddings.
4. Latent Diffusion: A UNet-based LDM de-noises a random latent over $T$ inference steps. Unlike most LDMs, the AudioLDM 2 UNet utilizes two sets of cross-attention embeddings: those from GPT2 and those from Flan-T5.
5. Decoding: The final de-noised latents are passed through a VAE decoder to recover a Mel spectrogram, which is then converted into an audio waveform by a vocoder.

Available Model Checkpoints

Checkpoint	Task	Model Size	Training Data (hours)
cvssp/audioldm2	Text-to-audio	1.1B	1150k
cvssp/audioldm2-music	Text-to-music	1.1B	665k
cvssp/audioldm2-large	Text-to-audio	1.5B	1150k

Inference Optimizations for Speed

To address the slow inference speeds of the original implementation, Hugging Face identifies four primary optimization techniques available in the diffusers library:

1. Flash Attention (SDPA)

By using PyTorch 2.0 or higher, the diffusers library automatically enables torch.nn.functional.scaled_dot_product_attention (SDPA). This provides a memory-efficient attention operation similar to Flash Attention, reducing computation time without changing the output quality.

2. Half-Precision (float16)

Converting model weights and computations from float32 to float16 (half-precision) significantly reduces GPU memory usage and improves inference speed with imperceptible changes to audio quality. This is achieved by passing torch_dtype=torch.float16 during the .from_pretrained call.

3. Torch Compile

Wrapping the UNet—the most computationally expensive part of the pipeline—with torch.compile(pipe.unet, mode="reduce-overhead", fullgraph=True) provides a substantial speed-up. While the first inference run is slow due to compilation overhead (potentially up to 2 minutes), all subsequent generations are significantly faster.

4. Efficient Schedulers

Replacing the default DDIMScheduler (which typically requires 200 steps) with a more performant scheduler like DPMSolverMultistepScheduler allows the model to achieve similar quality in only 20-25 inference steps. This optimization, combined with the others, can reduce the generation time of a 10-second sample to under 1 second.

Memory Management and Long-Form Audio

Generating long audio samples (e.g., 150 seconds) or using larger checkpoints (like audioldm2-large) increases the width of latent variables, which can lead to CUDA Out-of-Memory (OOM) errors because cross-attention memory scales quadratically with sequence length.

To mitigate this, Hugging Face recommends CPU offloading via pipe.enable_model_cpu_offload(). This technique keeps only the currently active model component on the GPU and offloads the rest to the CPU, allowing for the generation of long-form audio and the use of larger models on GPUs with limited RAM with a minimal penalty to inference time.