dnhkng/GLaDOS

This is the Personality Core for GLaDOS, the first steps towards a real-life implementation of the AI from the Portal series by Valve.

What it solves

It transforms a standard LLM into a proactive, multi-modal AI persona (specifically modeled after GLaDOS from Portal) that can see, hear, and speak without needing a wake word. It solves the problem of stilted, reactive voice assistants by implementing a low-latency pipeline and an autonomous "tick" loop that allows the AI to initiate conversation based on environmental triggers.

How it works

The project uses a multi-agent architecture inspired by Minsky's "Society of Mind," where specialized subagents (handling vision, memory, emotion, and research) contribute to a dynamic context. A main agent then processes this context to generate responses.

Key technical components include:

  • Input Pipeline: Uses Silero VAD for voice detection and Parakeet ASR for speech-to-text.
  • Vision: Employs FastVLM for scene understanding and change detection.
  • Autonomy: A background loop triggers the AI to speak based on time, vision events, or task updates.
  • Personality: Combines the PAD (Pleasure-Arousal-Dominance) model for reactive moods and HEXACO traits for a stable character.
  • Tooling: Integrates the Model Context Protocol (MCP) for home automation and system information.
  • Output: Uses a custom-trained TTS model (Kokoro) optimized for low latency (under 600ms round-trip).

Who it’s for

AI enthusiasts and hobbyists who want to build an embodied, autonomous AI assistant with a distinct personality, as well as developers interested in low-latency voice/vision pipelines and multi-agent orchestration.

Highlights

  • Proactive Interaction: No wake word required; the AI observes and speaks when it has something to say.
  • Low Latency: Optimized pipeline designed to keep response times under 600ms for natural flow.
  • Multi-Modal: Integrated camera (VLM) and microphone (ASR) for environmental awareness.
  • Extensible Tools: MCP support for connecting the AI to system stats and external home automation.
  • Emotional Engine: Dynamic mood shifts based on the PAD model and persistent personality traits.

Related

  • Project
    GetStream/Vision-Agents

    A framework for building low-latency, multi-modal AI agents that can watch, listen, and understand video in real time using a combination of CV models and LLMs.

  • Project
    huggingface/speech-to-speech

    Speech‑to‑Speech is a Hugging Face open‑source pipeline that turns spoken input into spoken output. It chains VAD → STT → LLM → TTS, each component being swappable (Silero VAD, Parakeet/Faster‑Whisper/Whisper STT, any OpenAI‑compatible LLM or local Transformers/MLX model, Qwen3‑TTS or other TTS back‑ends). The whole system speaks the OpenAI Realtime protocol over WebSocket/WebRTC, so existing OpenAI Agents SDK code works. Install with a single `pip install speech-to-speech`, then run `speech-to-speech serve` (server) and `speech-to-speech talk` (client) or `speech-to-speech local` (both together). It supports local‑only operation, mixed local/hosted LLMs, Docker deployment, and optional extras for alternative TTS/STT models. Designed for voice‑assistant robots (e.g., Reachy Mini) and low‑latency spoken AI applications.

  • Project
    beilusaiying/always-accompany

    A multi-purpose AI agent framework focused on context and attention mechanisms, providing layered memory and dynamic retrieval for long-term companionship, coding, and work.

  • Project
    hello-diana/MASCOT

    A multi-agent framework for building socio-collaborative AI companions that prevents persona collapse and social sycophancy through bi-level optimization of individual speakers and a coordinating director.

  • Project
    TEN-framework/ten-framework

    An open-source framework for building real-time multimodal conversational AI agents with low-latency audio, text, and visual capabilities.