MoonInTheRiver/DiffSinger

DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism (SVS & TTS); AAAI 2022; Official code

What it solves

DiffSinger is designed to solve the challenge of high-quality singing voice synthesis (SVS) and text-to-speech (TTS) generation. It aims to provide a more natural and expressive singing voice by utilizing a diffusion-based approach to generate audio waveforms from lyrics and pitch information.

How it works

The project implements a "shallow diffusion mechanism" to generate mel-spectrograms from inputs such as lyrics, MIDI, or ground-truth F0 (pitch) data. Depending on the pipeline used, it can predict pitch explicitly or implicitly. The generated mel-spectrograms are then converted into audible waveforms using vocoders like HiFiGAN or NSF-HiFiGAN.

Who it’s for

Researchers and developers working in audio synthesis, generative AI for music, and speech synthesis (TTS) who want to implement or experiment with diffusion-based audio generation.

Highlights

  • Dual Capability: Supports both Singing Voice Synthesis (SVS) and Text-to-Speech (TTS).
  • Flexible Pipelines: Offers multiple configurations for handling pitch (explicit vs. implicit prediction) and input types (MIDI, lyrics, and ground-truth F0).
  • Acceleration: Includes a plug-in for PNDM (Pseudo Numerical Methods for Diffusion Models) to accelerate the synthesis process.
  • Integration: Uses industry-standard vocoders like HiFiGAN and NSF-HiFiGAN for high-fidelity audio output.

Related

  • Project
    fishaudio/fish-diffusion

    A diffusion-based training framework for Text-to-Speech, Singing Voice Synthesis, and Singing Voice Conversion that supports multi-speaker training and efficient hardware utilization.

  • Project
    prophesier/diff-svc

    A singing voice conversion tool that uses diffusion models to transform input singing voices into a target timbre with basic pitch correction support.

  • Project
    sp-uhh/sgmse

    A PyTorch implementation of diffusion-based generative models for speech enhancement and dereverberation, removing noise and reverberation from audio signals.

  • Project
    SWivid/F5-TTS

    F5-TTS is a high-performance text-to-speech system using a Diffusion Transformer and flow matching to generate fluent, faithful speech with fast training and inference.

  • Project
    yxlllc/DDSP-SVC

    An efficient singing voice conversion project that enables high-quality AI voice changing with significantly lower hardware requirements and faster training times than traditional SVC models.