katipally/openlive

OpenLive – On‑device voice & vision layer for AI agents

What it is – OpenLive is a desktop application that turns any LLM‑or‑coding‑agent (Claude, OpenAI, Gemini, Ollama, etc.) into a spoken, visual assistant that runs entirely on your own computer. It wires together:

• Voice activity detection (Silero VAD)
• Streaming speech‑to‑text (Whisper)
• Turn‑taking logic (Smart‑Turn)
• Text‑to‑speech (Kokoro or Supertonic, with optional zero‑shot voice cloning)
• Optional camera / screen capture for vision‑enabled models. All of the audio processing happens locally on WebGPU; only the final transcript (and any optional image frames) are sent to the chosen model via its normal API key.

Key features

• Model‑agnostic – works with any provider you have a key for (Anthropic, OpenAI, Google, xAI, DeepSeek, Groq, Ollama, etc.).
• Agent‑agnostic – can drive coding agents that speak the Agent Client Protocol (Claude Code, Codex, Cursor, OpenCode, Hermes, …) as a child process.
• Voice cloning – record 5‑30 s of your own voice, generate a clone locally (ZipVoice), and have the assistant speak in that voice.
• Barge‑in – interrupt the assistant at any time; playback stops mid‑word.
• Vision support – send camera or screen frames each turn; a text‑only model can call a separate vision model.
• Floating mini‑mode – tiny always‑on‑top window with tray icon for quick access while you work.
• Privacy‑first – no audio is uploaded; API keys are stored encrypted (AES‑256‑GCM) and only the last four digits are displayed.
• Session persistence – conversations are saved as markdown, can be resumed from either the OpenLive UI or the agent’s CLI, and include cost/plan checklists.

Typical workflow

1. Install the app (macOS, Windows, Linux binaries are provided).
2. Add your model API key or install a coding‑agent CLI from Settings → Agents.
3. (Optional) record a short voice sample to clone your voice.
4. Start a “call”; the app listens, converts speech to text, sends the turn to the selected brain, streams the brain’s reply back to the TTS engine, and speaks it while optionally showing camera/screen frames.
5. Answer any permission prompts from the agent by voice or tap, and watch the plan/cost UI update in real time.

Getting started

• Binary – download the latest release for your OS, run the installer, paste your model key, and hit Start Call.
• From source –

  pnpm install
  pnpm desktop:dev   # launches Electron + local servers
  

  You can also run the web UI with pnpm dev (localhost:3000) and run tests via pnpm test.

Architecture at a glance

mic → VAD → streaming STT → end‑of‑turn → your AI (via API or ACP) → streaming TTS → speaker
                ↑
                └─ camera/screen frames (optional)

All components except the external model run in the Electron renderer using WebGPU. A lightweight WebSocket server mediates the turn and drives the ACP‑compatible coding agents.

Repo layout

• apps/desktop – Electron shell, tray, mini‑mode.
• apps/web – Next.js UI + on‑device voice engine.
• services/agent – WebSocket, ACP driver, voice‑cloning logic.
• packages/* – shared types, model adapters, encrypted JSON store.
• docs/ARCHITECTURE.md – deeper dive into the pipeline.

License – MIT (free to use, modify, and redistribute).

---

OpenLive is a genuine, open‑source project that provides the missing voice and vision plumbing for AI agents, letting you interact with LLMs or coding assistants entirely on‑device without recurring audio fees.