xzf-thu/VoiceMem

Infrastructure for the next generation of voice agents, designed to provide universal memory. It is divided into a left brain and a right brain, storing information and emotions respectively, while a fully streaming architecture eliminates latency at the fundamental level.

What it solves

VoiceMem provides a long-term memory system specifically designed for real-time voice agents. It solves the problem of high latency and high token costs associated with traditional memory retrieval, while enabling agents to remember not just factual information, but also the user's personality, emotions, and relationships.

How it works

VoiceMem uses a "streaming dual-brain" architecture to separate and process memory:

  • Left Brain: Manages factual information organized by schemas and entities for high-precision retrieval.
  • Right Brain: Handles emotional intelligence, personality, and relationship data using independent and cross-entity nodes.

The system operates in a streaming pipeline where audio is segmented, transcribed, and analyzed (ASR, speaker ID, emotion detection) while the user is still speaking. It uses speculative prefetching (0–300 ms) to retrieve relevant memories before the user finishes their sentence, injecting only the Top-K most relevant memories into the model's context to minimize token usage.

Who it’s for

Developers building real-time voice AI agents who need their models to have persistent, low-latency, and emotionally aware long-term memory without bloating the context window.

Highlights

  • Low Latency: Response times of 134 ms, significantly faster than alternatives like Mem0.
  • High Accuracy: Achieves 91.2% on the LoCoMo benchmark using only Top-5 memories.
  • Multimodal Memory: Capable of remembering speech, speakers, sound events, and music from audio.
  • Token Efficient: Uses approximately 430 memory tokens per query compared to thousands in other systems.
  • Integrated Pipeline: Bundles ASR, speaker identification, and emotion perception into the memory ingest process.

Related

  • Project
    mkiol/dsnote

    A privacy-focused note-taking app for Linux and Sailfish OS that provides offline speech-to-text, text-to-speech, and machine translation.

  • Project
    codeaholicguy/ai-devkit

    AI DevKit is a local npm‑based control plane that unifies configuration, monitoring, cross‑agent messaging, searchable SQLite memory, and composable engineering “skills” (requirements, design, testing, review, etc.) for many AI coding assistants such as Claude Code, Gemini CLI, Codex CLI, Cursor, and GitHub Copilot. It lets teams manage multiple agents from a single console, keep project conventions in a searchable store, and enforce a senior‑engineer workflow without modifying the underlying LLMs.

  • Project
    DSH-EAC/DSH-Desktop-EAC

    DSH‑Desktop‑EAC is a native desktop client that bundles DeepSeek Harness (the open‑source AI‑agent framework) with a full Node.js runtime, a plugin marketplace, auto‑updates, and UI customisation. It lets users run multi‑agent LLM workflows with a double‑click, sharing the same `dsh` core and data as the CLI while adding safety features like plugin snapshots and rollback.

  • Project
    qufei1993/skills-hub

    Skills Hub is a Tauri‑based desktop app that centralises AI‑coding “skills” (prompt/agent bundles) and syncs them via symlinks or copies to the global or project‑level skill folders of many AI coding tools (Cursor, Claude Code, Codex, etc.). It offers explore‑install, tagging, bulk actions, a recycle bin, automatic Git‑based device sync, and scheduled updates, all configurable through a multilingual UI.