Stability-AI/stable-audio-metrics

Metrics for evaluating music and audio generative models – with a focus on long-form, full-band, and stereo generations.

What it solves

It provides a standardized way to evaluate the quality of music and audio generative models, specifically focusing on long-form, full-band stereo audio. It ensures fair comparisons between different models by handling resampling and mono/stereo conversion automatically.

How it works

The project implements three primary evaluation metrics based on established models:

  • Fréchet Distance (at 48kHz) using OpenL3.
  • Kullback–Leibler (KL) divergence (at 32kHz) using PaSST.
  • CLAP score (at 48kHz) using CLAP-LAION.

These metrics can process audio files of variable lengths and are designed to run on GPUs for efficiency.

Who it’s for

Researchers and developers building generative audio and music AI models who need to quantitatively measure their output quality against benchmarks like MusicCaps, AudioCaps, and Song Describer.

Highlights

  • Standardized Benchmarking: Supports evaluation against the MusicCaps, AudioCaps, and Song Describer datasets.
  • Flexible Input: Handles variable-length audio inputs and manages stereo/mono differences.
  • Pre-computed Statistics: Includes "no-audio" examples that use pre-computed reference statistics and embeddings to speed up evaluation.
  • GPU Optimized: Specifically designed for GPU execution to avoid slow CPU processing.

Related

  • Project
    wavlab-speech/versa

    A comprehensive toolkit providing a unified interface to over 90 metrics for evaluating the quality, intelligibility, and technical properties of speech and audio.

  • Dispatch
    Big Bench Audio Release

    Artificial Analysis has released Big Bench Audio, a dataset of 1,000 audio questions designed to evaluate the reasoning capabilities of audio language models, revealing a significant performance gap between text and speech reasoning.

  • Project
    AudioLLMs/AudioBench

    A universal benchmark for evaluating Audio Large Language Models (AudioLLMs) across 50+ datasets covering speech, audio-scene, and voice understanding.

  • Project
    Karine-Huang/T2I-CompBench

    A comprehensive benchmark and evaluation suite for text-to-image generation models, focusing on their ability to handle complex compositional prompts, attribute binding, and spatial relationships.

  • Dispatch
    HELMET: Holistically Evaluating Long-context Language Models

    Hugging Face and Princeton researchers introduced HELMET, a comprehensive benchmark designed to replace synthetic tests like needle-in-a-haystack with diverse, controllable, and reliable real-world evaluations for long-context language models.