NVIDIA-NeMo/Speech

A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)

What it solves

NVIDIA NeMo Speech provides a comprehensive framework for researchers and developers to create, customize, and deploy advanced speech AI models. It simplifies the process of building systems for automatic speech recognition (ASR), text-to-speech (TTS), and speech-based large language models (Speech LLMs), reducing the need to write everything from scratch by providing pre-trained checkpoints and reusable code.

How it works

Built on PyTorch, NeMo Speech allows developers to leverage existing model architectures and open-weight checkpoints (available via HuggingFace) to build speech applications. It supports a wide range of capabilities including streaming ASR with controllable latency, multilingual TTS, and full-duplex, interruptible voice conversations. The toolkit is designed to be flexible, allowing users to install it over their existing Python/PyTorch/CUDA stack or use optimized NVIDIA NGC containers for turnkey deployment.

Who it’s for

It is primarily designed for AI researchers and PyTorch developers who are building professional-grade speech models or integrating speech capabilities into larger AI systems.

Highlights

  • Diverse Speech Modalities: Supports Automatic Speech Recognition (ASR), Text-to-Speech (TTS), and Speech LLMs.
  • High-Performance Streaming: Features models like Nemotron-3.5-ASR-Streaming with controllable latency (80ms-1s) and high concurrency.
  • Multilingual Support: Includes models like MagpieTTS and Parakeet/Canary with support for numerous global languages.
  • Advanced Conversational AI: Includes Nemotron 3 VoiceChat for natural, low-latency, full-duplex conversations.
  • Flexible Deployment: Offers multiple installation paths, including uv for reproducible stacks and Docker containers for optimized GPU performance.

Related

  • Project
    istupakov/onnx-asr

    A lightweight Python package for Automatic Speech Recognition using ONNX models, eliminating the need for PyTorch or Transformers for fast deployment on edge and server hardware.

  • Project
    yeyupiaoling/MASR

    A PyTorch-based automatic speech recognition framework that supports streaming and non-streaming inference across multiple model architectures and languages.

  • Project
    NVIDIA-NeMo/Automodel

    NeMo AutoModel is a Python library from NVIDIA that provides ready‑to‑use, highly‑optimized training recipes for a huge range of LLMs, VLMs, and diffusion models. It supports full‑parameter and LoRA fine‑tuning, multi‑node tensor‑ and pipeline‑parallelism, and even speculative‑decoding drafters, all driven by simple YAML configs.

  • Project
    speechbrain/speechbrain

    An open-source PyTorch toolkit for Conversational AI that provides recipes and tools for speech and text processing, including speech recognition and synthesis.

  • Project
    NVIDIA-NeMo/RL

    NeMo RL is NVIDIA’s open‑source library for scalable reinforcement‑learning fine‑tuning of large language and vision‑language models. It supports a range of RL algorithms (GRPO, DAPO, GDPO, etc.), works with both PyTorch DTensor and Megatron Core backends, and offers high‑performance generation via vLLM or Megatron inference. Ready‑to‑run Docker containers, Ray‑based distributed training, and extensive YAML recipes make it suitable for quick prototyping up to multi‑node, multi‑GPU production runs.