OpenAI MuseNet

OpenAI has introduced MuseNet, a deep neural network that generates musical compositions up to four minutes long featuring 10 different instruments. The model can blend diverse musical styles—ranging from Mozart and Chopin to the Beatles and country music—without being explicitly programmed with music theory, instead discovering patterns by predicting the next token in a large dataset of MIDI files.

Architecture and Technical Foundation

MuseNet is built on a large-scale transformer model using the same general-purpose unsupervised technology as GPT-2. It specifically utilizes the recompute and optimized kernels of the Sparse Transformer to support a 72-layer network with 24 attention heads.

Key technical specifications include:

  • Context Window: Full attention over a context of 4,096 tokens, which allows the model to maintain long-term musical structure and melodic consistency.
  • Training Objective: The model is trained to predict the next token in a sequence, whether the data is audio or text.
  • Domain Application: Music serves as a testing ground for the Sparse Transformer because it bridges the gap between text (fluid token structure) and images (fixed spatial relationships), making structural errors in rhythm or melody highly audible.

Control Mechanisms: Composer and Instrumentation Tokens

To provide users with control over the output, OpenAI implemented composer and instrumentation tokens. These tokens are prepended to samples during training, allowing the model to associate specific styles or instruments with the resulting note predictions.

At generation time, these tokens act as conditioners. For example, a prompt can specify a "Rachmaninoff piano start" or a combination of "the band Journey, with piano, bass, guitar, and drums." While these tokens act as strong suggestions, they are not strict requirements; MuseNet calculates probabilities across all possible notes and instruments, meaning it may occasionally deviate from the requested instrumentation.

Dataset and Tokenization

MuseNet was trained on hundreds of thousands of MIDI files sourced from ClassicalArchives, BitMidi, the MAESTRO dataset, and various online collections covering jazz, pop, African, Indian, and Arabic styles.

Token Encoding

OpenAI experimented with several encoding methods before settling on a concise format that combines pitch, volume, and instrument information into a single token. Other attempted methods included:

  • Chordwise approach: Assigning a single token to every combination of notes sounding simultaneously.
  • Condensed patterns: Focusing only on note starts and using byte pair encoding.
  • Timing markers: Testing both tempo-scaled tokens (musical beats) and absolute time in seconds.

Training Augmentations

To improve robustness, the training process included:

  • Pitch Transposition: Raising and lowering pitches (reduced in later stages to keep instruments within their natural ranges).
  • Volume Augmentation: Adjusting overall volumes of samples.
  • Timing Augmentation: Slightly speeding up or slowing down pieces when using absolute time encoding.
  • Mixup: Applying mixup in the token embedding space.
  • Inner Critic: A discriminator trained to distinguish between actual dataset samples and the model's own past generations, used to select higher-quality samples during generation.

Structural Embeddings

Beyond standard positional embeddings, MuseNet employs several specialized embeddings to provide structural context:

  • Timing Embeddings: A learned embedding that tracks the passage of time, ensuring notes sounding simultaneously share the same timing embedding.
  • Chord Embeddings: An embedding for each note within a chord to help the model learn relative attention between notes.
  • Global Structural Embeddings: Two embeddings that orient the model within the larger piece: one dividing the piece into 128 parts and another acting as a countdown from 127 to 0 as the model approaches the end token.

Sources