lhotse-speech/lhotse

Tools for handling multimodal data in machine learning projects.

What it solves

Lhotse simplifies the complex process of preparing multimodal data (speech, audio, video, image, and text) for machine learning. It removes the need for manual, rigid data pipelines by providing flexible tools to handle metadata, data augmentation, and efficient loading for large-scale training.

How it works

Lhotse uses a "cut-based" approach where audio or video recordings are represented as "cuts"—virtual segments that can be mixed, truncated, or padded on-the-fly without needing to store every variation on disk. It manages metadata using human-readable text manifests and integrates directly with PyTorch through task-specific Dataset classes. For high-performance I/O, it supports sequential formats like Lhotse Shar and WebDataset to optimize data loading during distributed multi-node training.

Who it’s for

It is designed for researchers and developers working on speech processing, multimodal AI, and large-scale audio/video model training who need a Python-centric way to manage massive datasets.

Highlights

  • Multimodal Support: Handles audio, text, image, and video modalities.
  • Efficient Data Loading: Features state-of-the-art algorithms for dataset blending and on-the-fly bucketing.
  • Flexible Manipulation: Allows for on-the-fly mixing, truncation, and padding of data cuts to minimize storage.
  • PyTorch Integration: Provides seamless integration with PyTorch via custom Dataset classes and samplers.
  • Optimized Storage: Supports sequential I/O formats (Lhotse Shar) and compressed feature storage (lilcom) for better efficiency.

Related

  • Project
    facebookresearch/multimodal

    A PyTorch library for training state-of-the-art multimodal models at scale, providing modular building blocks and pretrained weights for models like CLIP and BLIP-2.

  • Project
    iver56/audiomentations

    A Python library for audio data augmentation that provides a wide variety of transforms to make audio deep learning models more robust for real-world use.

  • Project
    facebookresearch/stopes

    A library for preparing machine translation data, providing tools for cleaning monolingual data, mining bitext, and running knowledge distillation pipelines.

  • Project
    mosaicml/streaming

    A high-performance data streaming library for PyTorch that enables fast, deterministic loading of massive datasets from cloud storage during distributed AI training.

  • Project
    pytorch/audio

    A PyTorch-based audio library that provides tools for processing audio data for machine learning, featuring GPU acceleration and specialized audio transforms.