meta-pytorch/torchcodec

What it solves

TorchCodec provides a PyTorch-native way to decode and encode media files (videos, audio, and images) directly into PyTorch tensors. It removes the complexity of manually handling media codecs and the FFmpeg command-line tool, allowing researchers and developers to efficiently feed media data into ML models for training or inference.

How it works

TorchCodec abstracts the complexity of FFmpeg for video and audio processing, while providing its own image decoders and encoders. It offers Pythonic APIs that mirror PyTorch conventions, enabling users to perform time-based indexing or frame-based indexing of media files. The library supports both CPU and CUDA GPU acceleration for decoding and encoding, ensuring that data is returned as tensors ready for immediate use in PyTorch transforms.

Who it’s for

It is designed for ML engineers and researchers working with multimodal data, specifically those using PyTorch to train or run inference on video, audio, or image datasets.

Highlights

• PyTorch Integration: Returns data as tensors, making it seamless to integrate with the PyTorch ecosystem.
• Hardware Acceleration: Supports decoding and encoding on both CPU and CUDA GPUs (including NVDEC/NVENC).
• Flexible Indexing: Supports simple frame indexing, time-based indexing (PTS), and batching of frames.
• Broad Format Support: Leverages FFmpeg for wide coverage of audio/video formats and supports JPEG, PNG, WebP, GIF, AVIF, and HEIC for images.