OpenAI Jukebox

OpenAI Jukebox is a neural network that generates music as raw audio, enabling the synthesis of various genres and artist styles, including rudimentary singing. Unlike symbolic music generators that produce MIDI-like notes, Jukebox models audio directly, allowing it to capture human voices and complex timbres.

Technical Approach: Hierarchical Compression and Generation

Jukebox overcomes the challenge of modeling raw audio—where a four-minute CD-quality song contains over 10 million timesteps—by using a hierarchical autoencoder and transformer-based priors.

Raw Audio Compression via VQ-VAE

Jukebox uses a Vector Quantized-Variational AutoEncoder (VQ-VAE) to compress 44kHz raw audio into a discrete latent space. The model employs three levels of compression (8x, 32x, and 128x) with a codebook size of 2048 for each level. To improve performance, OpenAI implemented three specific modifications:

  • Random Restarts: To prevent codebook collapse, codebook vectors are randomly reset to encoded hidden states if their usage falls below a certain threshold.
  • Separate Decoders: Each level uses independent decoders to reconstruct the input, maximizing the utility of upper levels.
  • Spectral Loss: A spectral loss function is added to penalize the difference between input and reconstructed spectrograms, aiding the reconstruction of higher frequencies.

Music Generation via Sparse Transformers

Once the audio is compressed, three levels of autoregressive prior models (based on a simplified variant of Sparse Transformers) learn the distribution of these codes:

  • Top-level Prior: Models long-range structure and high-level semantics (melodies and singing) over a context of 8,192 codes (approx. 24 seconds of audio).
  • Upsampling Priors: Two subsequent levels add local musical structures and timbre, refining the audio quality over shorter contexts (6 seconds and 1.5 seconds, respectively).

Conditioning and Dataset

Jukebox is trained on a curated dataset of 1.2 million songs, including 600,000 in English, paired with metadata and lyrics from LyricWiki. The model supports three types of conditioning to steer generation:

  • Artist and Genre: By providing artist and genre labels, the model reduces prediction entropy and can generate music in specific styles. Unsupervised clustering (via t-SNE) shows the model naturally groups similar artists and genres together.
  • Lyrics: To handle unaligned lyrics, Jukebox uses a linear alignment heuristic and, for faster genres like hip hop, employs Spleeter to extract vocals and NUS AutoLyricsAlign for word-level alignment. An encoder-decoder attention layer allows the music decoder to attend to the lyrics encoder.

Current Limitations

Despite its capabilities, Jukebox has several technical and creative gaps compared to human-composed music:

  • Lack of Large-Scale Structure: While local coherence and chord patterns are present, the model does not yet generate repeating larger structures like choruses.
  • Audio Artifacts: The downsampling and upsampling process introduces discernible noise.
  • Sampling Speed: Due to its autoregressive nature, the model is slow; it takes approximately 9 hours to render one minute of audio.
  • Dataset Bias: Training is currently limited primarily to English lyrics and Western music.

Development Timeline

  • July 2019: Initial raw audio model developed to recreate instruments like piano and violin.
  • September 2019: Expanded dataset with artist and genre labels, enabling full-length songs with long-range coherence.
  • January 2020: Scaled VQ-VAE to 44kHz and the top-level prior from 1B to 5B parameters, resulting in clearer singing and higher quality.
  • January 2020: Introduced lyrics conditioning to learn pronunciation and singing alignment.

Sources