xzf-thu/VoiceMem

Infrastructure for the next generation of voice agents, designed to provide universal memory. It is divided into a left brain and a right brain, storing information and emotions respectively, while a fully streaming architecture eliminates latency at the fundamental level.

What it solves

VoiceMem provides a long-term memory system for real-time voice agents, allowing them to remember factual information about users, their emotional states, and personality traits over time. It addresses the high latency and high token cost typically associated with retrieving large amounts of memory in voice interactions.

How it works

VoiceMem uses a "streaming dual-brain" architecture to separate memory types:

  • Left Brain: Manages factual information using schemas and entities for precise retrieval.
  • Right Brain: Manages emotional intelligence, personality, and relationship information through independent and cross-entity nodes.

The system operates in a streaming fashion, performing ASR (Automatic Speech Recognition), memory extraction, and retrieval while the user is still speaking. It uses speculative prefetching (0–300 ms) and hierarchical storage to minimize latency, only injecting the top-K most relevant memories into the model's context to keep token usage low.

Who it’s for

Developers building real-time voice AI agents who need their bots to have a persistent, emotional, and personality-aware memory of their users without sacrificing response speed.

Highlights

  • Low Latency: Response times around 134 ms, significantly faster than traditional memory engines.
  • Emotional Awareness: Tracks not just what was said, but who the user is and how they feel.
  • Efficient Token Use: Uses approximately 430 memory tokens per query compared to thousands in other systems.
  • Multimodal Memory: Capable of remembering voiceprints, speaker IDs, sound events, and music from audio.
  • Open Source: Includes a family of compatible models (Qwen2.5-Omni, Qwen3-Omni, Step-Audio2-Mini) and a training dataset (ChatMem-400K).

Related

  • Project
    TeleAI-UAGI/telemem

    TeleMem is a high-performance, drop-in memory management layer for AI agents that provides character-specific memory profiles and a full pipeline for storing and reasoning over video content.

  • Project
    aiming-lab/SimpleMem

    A unified memory stack for LLM agents that provides semantically lossless compression for text and multimodal data, featuring a self-evolving retrieval system to maximize recall while minimizing token usage.

  • Project
    VictorTaelin/OptMem

    A permanent memory system for AI agents that uses a binary tree of summaries and a CLI tool to maintain persistent knowledge across sessions.

  • Project
    zjunlp/LightMem

    LightMem is a lightweight memory management framework for LLMs and AI agents, providing efficient storage, retrieval, and update mechanisms. It reduces token usage and latency while maintaining high accuracy on long-context benchmarks.

  • Project
    fuyuxiang/echo-agent

    Echo Agent is an open‑source, self‑hosted AI‑agent runtime that provides long‑term memory, automatic skill extraction/evolution, and pluggable tool integration. It can run interactively in a terminal, as a long‑living Gateway service with a web dashboard and API, or be embedded in a desktop application (EchoAgent Desktop). The core runtime manages the agent loop, session context, model routing, permissions, and automation channels, while the memory system recalls past facts and the evolution system turns execution traces into reusable Skills. Install via `pip install "echo-agent[all]"`, configure an LLM API key, and start with `echo-agent run` or `echo-agent gateway`.