meta-pytorch/torchcodec

What it solves

TorchCodec provides a PyTorch-native way to decode and encode media files (videos, audio, and images) directly into PyTorch tensors. It removes the complexity of using FFmpeg manually and ensures that media data is ready for machine learning models without cumbersome conversion steps.

How it works

The library abstracts the complexity of FFmpeg for video and audio processing, while providing dedicated decoders for images. It offers Pythonic APIs that mirror PyTorch conventions, allowing users to load media as tensors on either CPU or CUDA GPUs. It supports time-based indexing and metadata extraction for video streams.

Who it’s for

ML engineers and researchers who need to efficiently load and process media data for training or inference in PyTorch-based workflows.

Highlights

• PyTorch Integration: Returns data as tensors, making it compatible with PyTorch transforms and model training.
• Hardware Acceleration: Supports decoding and encoding on both CPU and CUDA GPUs (including NVDEC/NVENC).
• Flexible Indexing: Offers simple indexing, time-based indexing, and PTS (Presentation Time Stamp) retrieval for video frames.
• Broad Format Support: Handles various image formats (JPEG, PNG, WebP, GIF, AVIF, HEIC) and video/audio via FFmpeg.