Deep-unlearning/smol-audio

What it solves

It provides practical, hands-on guides for optimizing and customizing audio AI models, specifically focusing on shrinking and fine-tuning them for specialized tasks like automatic speech recognition (ASR) and audio captioning.

How it works

The project consists of a collection of Jupyter notebooks that demonstrate how to use the Hugging Face ecosystem to fine-tune various audio models. It covers different techniques including full fine-tuning and Low-Rank Adaptation (LoRA) across several model architectures such as Whisper, Granite Speech, Audio Flamingo 3, Parakeet, and Voxtral.

Who it’s for

Developers and AI practitioners who want to adapt existing audio models to custom languages, specific domains, or new tasks like audio-video retrieval and classification.

Highlights

• Support for ASR fine-tuning across multiple models (Whisper, Granite Speech, Parakeet, Voxtral).
• Guides for audio captioning using Audio Flamingo 3.
• Implementation of both full fine-tuning and LoRA for efficient model customization.
• Examples of zero-shot video classification and audio-text retrieval using Meta's Perception Encoder for Audio-Video (PE-AV).