Liquid4All/liquid-audio

What it solves

Liquid Audio provides a lightweight, end-to-end audio foundation model designed for low-latency, real-time speech-to-speech conversations. It eliminates the need for separate pipeline components by integrating speech recognition and synthesis into a single model, making it suitable for resource-constrained devices.

How it works

Built on the LFM2 backbone, the model processes audio and text as tokens. It uses two distinct generation modes to optimize performance:

• Interleaved Generation: Outputs text and audio tokens in a fixed pattern, minimizing the time to first audio output for fluid, real-time conversations.
• Sequential Generation: The model decides when to switch modalities via special tokens, which is better suited for non-conversational tasks like Automatic Speech Recognition (ASR) and Text-to-Speech (TTS).

Who it’s for

Developers building real-time voice assistants, speech-to-text/text-to-speech applications, or those looking to fine-tune a small-scale audio foundation model for specific tasks like function calling or specialized chat.

Highlights

• End-to-End Architecture: Supports speech-to-speech, ASR, and TTS within one model.
• Low Latency: Optimized for real-time interaction on constrained hardware.
• Flexible Generation: Offers both interleaved and sequential modes for different use cases.
• Fine-tuning Support: Includes data preparation tools and a lightweight trainer for custom task adaptation.
• Multilingual Support: Includes a dedicated Japanese version of the model.