ShayneP/local-voice-ai

What it solves

Local Voice Agent is a private, low-latency voice assistant designed to run entirely on your own hardware. It eliminates the need for cloud-based voice services by integrating speech recognition, a language model, and speech generation into a single supervised application.

How it works

The project uses a modular stack of local AI models to handle the voice interaction loop:

• Speech Recognition (STT): Uses Nemotron Q8 for streaming recognition (with Whisper as a manual fallback) to ensure low latency by sending partial transcripts while the user speaks.
• Language Model (LLM): Powered by llama.cpp to run local LLMs like Qwen3 or Gemma 4, depending on the hardware profile selected.
• Speech Generation (TTS): Uses Kokoro (or Kokoro ONNX) for local voice synthesis.
• Orchestration: Powered by LiveKit Agents to manage the real-time communication and streaming.

Who it’s for

Users who want a private voice assistant that works offline or on local networks, specifically those with NVIDIA GPUs, Apple Silicon, or NVIDIA Jetson Orin Nano devices.

Highlights

• Hardware-Aware Profiles: Automatic selection of model stacks (lean, compact, balanced) based on detected memory and device type.
• Low Latency: Streaming speech recognition via Nemotron allows for faster responses than traditional "wait-for-utterance" systems.
• Cross-Platform Support: Native setup for Linux CPU, NVIDIA Desktop, Apple Silicon, and Jetson Orin.
• Remote Client Mode: Allows the voice server to run on a powerful device (like a Jetson) while the user interface runs on a laptop browser.
• Flexible Configuration: Ability to replace local services with external APIs (e.g., LiveKit Cloud) via environment variables.