fluxions-ai/vui

Vui – Streaming Conversational Voice Assistant

What it is – Vui (pronounced “vooey”) is an open‑source, real‑time voice‑assistant stack. A single Python server glues together speech‑to‑text, a local large language model (LLM), and a streaming text‑to‑speech (TTS) model so you can talk to a computer and hear a spoken reply instantly.

Core model – Vui Nano – a 300 M parameter speech transformer built on the Qwen‑3 TTS codec. It was trained on conversational speech (including breaths, laughter, hesitations) and can be voice‑cloned to several preset personalities.

Key capabilities

• Full‑duplex voice loop – WebRTC + WebSocket pipeline: ASR → LLM → TTS, with VAD‑driven turn‑taking and barge‑in (you can interrupt a reply).
• Streaming TTS – ~9× real‑time on an RTX 4090, using bf16 inference and CUDA graphs.
• OpenAI Realtime API compatibility – drop‑in ws://…/v1/realtime endpoint for any client that follows OpenAI’s spec.
• One‑shot voice‑note endpoint – POST /v1/voice-note runs the whole pipeline in a single HTTP call (audio in, JSON‑with‑base64‑WAV out).
• Pluggable back‑ends – choose between Ollama, vLLM, or any OpenAI‑compatible service for the LLM; pick faster‑whisper (GPU) or Moonshine (CPU/ONNX) for ASR.
• Tool routing (“thoughts stream”) – a parallel LLM watches each turn and can invoke ~15 built‑in tools (memories, timers, web search, delegation to a Claude‑style task server) without a wake‑word grammar.
• Voice cloning & conditioning – upload a sample to clone a speaker; two conditioning vectors let you bias speech quality (six SQ channels) and words‑per‑second (WPS).
• Cross‑platform support – Docker‑compose for Linux + NVIDIA GPU, native install on macOS (MLX backend for Apple Silicon), and documented paths for mobile access via Cloudflare Tunnel or Tailscale.

Installation

# One‑liner installer (detects Docker vs native, pulls dependencies, downloads the model)
curl -fsSL https://install.fluxions.ai | bash

The script clones the repo to ~/vui, sets up Python (uv), installs ffmpeg, Ollama, and the Claude Code CLI, then launches the UI at http://localhost:8080.

Typical Docker workflow

1. Install Docker + NVIDIA Container Toolkit.
2. Pull an LLM for Ollama, e.g. ollama pull qwen3.5:4b.
3. docker compose up -d – brings up the Vui streaming container (and optional Ollama/Claude sidecars).
4. Open the browser, allow microphone access, and start talking.

Native workflow (no Docker) – install ffmpeg, run uv sync (or just uv sync on Apple Silicon), start Ollama, pull a model, then:

python -m vui.serving.stream   # server on http://localhost:8080

Environment variables (VUI_OLLAMA_URL, VUI_OLLAMA_MODEL, VUI_LLM_BACKEND, etc.) let you point the server at any compatible LLM backend.

Demo utilities

• demo.py – a Gradio UI that runs the TTS model alone (voice‑clone upload, CLI render).
• docker compose profiles let you add an Ollama container or a Claude‑task sidecar for delegated, multi‑step actions (email, calendar, web research).

Extensibility

• ASR – swap between faster‑whisper (GPU) and Moonshine (CPU/ONNX) from the UI.
• LLM – any Ollama model, vLLM server, or OpenAI‑compatible endpoint.
• Tools – the “thoughts” router is pluggable; you can add your own local functions (timers, smart‑home toggles) via the docs/thoughts-tools.md guide.
• Search providers – Serper, Brave, or Tavily APIs for single‑query factual look‑ups.

Where to learn more

• Live demo: https://fluxions.ai/talk
• Launch blog post (design notes & roadmap): https://fluxions.ai/blog/vui-launch
• Model on Hugging Face: https://huggingface.co/fluxions/vui
• Full API spec: docs/realtime-api.md
• Configuration details: docs/configuration.md
• Claude task‑server docs: docs/claude-task-server.md

Bottom line – Vui provides a production‑grade, end‑to‑end voice‑assistant pipeline that you can run locally (GPU or Apple Silicon) or in Docker, with interchangeable ASR/LLM/TTS components, tool integration, and an OpenAI‑compatible realtime interface. It is a genuine AI/ML project focused on speech‑centric conversational agents.