MidiTok

Summary MidiTok is a Python package that converts MIDI and abc music files into sequences of tokens. These tokens can be fed to machine‑learning models (e.g., Transformers) for music generation, transcription, or other MIR tasks.

Purpose Provide a unified interface for various music tokenization methods, enable training of tokenizers with BPE/Unigram/WordPiece, offer data‑augmentation utilities, and integrate with the Hugging Face Hub for sharing models.

How it works

• Uses Symusic to read/write MIDI and abc files.
• Tokenizers such as REMI take a Score object and output a list of token IDs.
• Tokenizers can be trained on a corpus of MIDI files (tokenizer.train) to learn a vocabulary via BPE, Unigram, or WordPiece.
• The trained tokenizer can then encode (tokenizer(midi)) and decode (tokenizer(tokens)) music, returning PyTorch, TensorFlow, or NumPy tensors.
• Helper utilities split long MIDI files into training‑size chunks, build PyTorch DatasetMIDI and DataCollator, and create a DataLoader for model training.

Key features

• Implements multiple tokenizations: REMI, REMI+, MIDI‑Like, TSD, Structured, CPWord, Octuple, MuMIDI, MMM, PerTok.
• Supports training tokenizers with BPE, Unigram, and WordPiece.
• Includes data‑augmentation methods.
• Hugging Face Hub integration (push_to_hub, from_pretrained).
• Works with PyTorch, TensorFlow, and NumPy tensors.
• Provides DatasetMIDI, DataCollator, and split_files_for_training utilities.

Usage

from miditok import REMI, TokenizerConfig
from symusic import Score

config = TokenizerConfig(num_velocities=16, use_chords=True, use_programs=True)
tokenizer = REMI(config)

midi = Score("path/to/your_midi.mid
tokens = tokenizer(midi)               # → token IDs
converted_back = tokenizer(tokens)     # → Score object

Training a tokenizer:

from pathlib import Path
files = list(Path("midis\)).glob("**/*.mid
tokenizer.train(vocab_size=30000, files_paths=files)
tokenizer.save(Path("tokenizer.json
tokenizer.push_to_hub("username/model-name", private=True, token="HF_TOKEN

Preparing data for PyTorch:

from miditok.utils import split_files_for_training
from miditok.pytorch_data import DatasetMIDI, DataCollator
from torch.utils.data import DataLoader

split_files_for_training(files_paths=files, tokenizer=tokenizer,
                         save_dir=Path("chunks\)), max_seq_len=1024)

dataset = DatasetMIDI(
    files_paths=list(Path("chunks\)).glob("**/*.mid\)),
    tokenizer=tokenizer,
    max_seq_len=1024,
    bos_token_id=tokenizer["BOS_None"],
    eos_token_id=tokenizer["EOS_None\)
)
collator = DataCollator(tokenizer.pad_token_id, copy_inputs_as_labels=True)
dataloader = DataLoader(dataset, batch_size=64, collate_fn=collator)

for batch in dataloader:
    # train your model on batch
    ...

Limitations / TODO

• Music‑XML support not yet implemented.
• Missing no_duration_drums option to drop duration tokens for drums.
• Control Change messages are not handled.
• Global/track event parsing could be sped up with Rust/C++ bindings.