mudler/parakeet.cpp

What it solves

parakeet.cpp is a high-performance C++ inference port of NVIDIA's NeMo Parakeet speech-recognition models. It removes the need for a Python runtime at inference time, providing a dependency-light way to run automatic speech recognition (ASR) quickly on both CPUs and GPUs.

How it works

Built on the ggml library, the project implements the inference logic for various Parakeet model families (CTC, RNNT, TDT, and hybrid TDT-CTC). It supports GGUF format models, which can be converted from NeMo checkpoints using provided scripts. The engine supports quantization (f16, q8_0, q4_k, etc.) to reduce memory usage and increase speed, and it includes a cache-aware streaming mode with end-of-utterance (EOU) detection for specific models.

Who it’s for

Developers and engineers who need to deploy NVIDIA Parakeet ASR models in production environments where Python is not desired, or where maximum CPU/GPU performance and low memory overhead are required.

Highlights

• High Performance: Faster than NeMo's PyTorch runtime on both CPU and GPU, with byte-identical transcripts.
• Broad Model Support: Covers all offline Parakeet families and supports multilingual, prompt-conditioned streaming models.
• Low Dependency: Written in C++17 with no Python requirement for inference.
• Flexible Deployment: Provides a CLI, an OpenAI-compatible HTTP server, and a C-API for direct integration into other applications.
• Hardware Acceleration: Supports CUDA, Metal, Vulkan, and CPU backends via ggml.
• Batching: Includes an opt-in batched decoding path to significantly increase throughput on GPUs.