fikrikarim/parlor

On-device, real-time multimodal AI with features similar to GPT-Live

What it solves

Parlor provides a fully on-device, real-time multimodal AI experience similar to GPT-Live. It allows users to interact with an AI via voice and camera without relying on cloud services, ensuring low latency and privacy while supporting complex interactions like barge-in (interrupting the AI) and hands-free turn-taking.

How it works

Parlor uses a cascade system to process multimodal inputs and generate responses:

  1. Input Processing: A browser-based frontend captures audio (PCM) and camera frames (JPEG), sending them via WebSockets to a FastAPI server.
  2. Turn Detection: Silero VAD handles initial silence detection, and smart-turn-v3 determines if the user has actually finished their thought to avoid premature interruptions.
  3. Core Model: Gemma 4 (via llama.cpp) processes the audio and visual data to generate a text reply.
  4. Speech Generation: Kokoro TTS converts the text into speech, streaming it sentence-by-sentence to the browser for immediate playback.
  5. Action Handling: A separate grammar-forced JSON head manages non-speech actions (like timers or mode switches) so that control markup doesn't leak into the spoken audio.
  6. Optional Research: If configured with an API key, a background reasoner can perform web research using a frontier model while the conversation continues.

Who it’s for

Developers and AI enthusiasts who want a high-performance, local multimodal AI assistant that runs on Apple Silicon (MacBook M3 Pro) or Linux with a GPU, and those who prefer self-hosted alternatives to proprietary voice AI services.

Highlights

  • Fully Local: Runs 100% on-device using Gemma 4 and Kokoro TTS.
  • Real-time Interaction: Supports barge-in and low-latency streaming of both transcripts and audio.
  • Hands-free: Uses advanced turn-detection to distinguish between mid-thought pauses and the end of a turn.
  • Specialized Modes: Includes a live translation mode for consecutive interpretation and a "just-listen" mode for silent scribing.
  • Integrated Tools: Built-in support for timers and background web research.

Related