Lynpoint/CyberVerse

Self hosted, real-time digital human agent platform. Build voice-first AI agents with WebRTC, persona memory, tools, RAG, and optional digital-human video.

What it solves

CyberVerse tackles the challenge of creating lifelike, real-time conversational AI characters that act like a digital human. Instead of a plain chatbot, you get an agent that can hold a live voice conversation, interrupt and be interrupted naturally, remember past sessions, recall character-specific knowledge, and optionally animate a face from just one photo. It also solves the performance problem of long-running tasks (search, research, report generation) by offloading them to background subagents so voice interactions stay snappy.

How it works

CyberVerse is a multi-agent framework built on WebRTC for low-latency real-time voice/video interaction. The core is a PersonaAgent that stays in the foreground, managing fluid conversation, interruptions, and context switches. Long-running tasks are delegated asynchronously to SubAgents, so the main character can keep answering while work happens in the background. Each character's conversation history is persisted locally and auto-loaded on return, and you can import knowledge sources (documents, bios) that get indexed for retrieval-augmented generation (RAG) to align answers with the persona.

For the digital-human video capability, CyberVerse integrates local models like FlashHead (1.3B) and LiveAct (18B) or cloud services (Baidu Xiling, Xunfei Digital Human). The stack is plugin-based: brain, voice, hearing, tools, memory, and face modules are all replaceable. Configuration lives in config/cyberverse.yaml, and model definitions for LLM, TTS, ASR, and embeddings auto-discover from infra/config/*_models/, with overrides under config/*_models/. A web UI at /settings lets you swap providers and models. The system can run in pure voice mode (no GPU for avatar) or with avatar video enabled, and includes a real-time performance factor (RTP) check to diagnose streaming lag.

Who it's for

Developers and tinkerers who want to build interactive AI characters, digital-human assistants, or J.A.R.V.I.S.-like companions. It requires familiarity with Node.js, Go, Conda/Python, and GPU setup if you enable avatar video, but cloud images are available for faster start. The project is GPL v3, so it also appeals to those who want a self-hostable, extensible alternative to closed digital-human platforms.

Highlights

  • Full-duplex real-time voice interaction via WebRTC with interruption support
  • Turns a single photo into a live digital-human video avatar (local models FlashHead/LiveAct or cloud APIs)
  • Multi-agent design: PersonaAgent for conversation + SubAgents for background tasks (search, research, report generation)
  • Persistent character memory and RAG over imported knowledge bases and documents
  • Plugin-based architecture with replaceable brain, voice, hearing, tools, memory, and face modules
  • Auto-discovery of model definitions and provider switching via web UI
  • Supports 100+ LLM providers through LiteLLM, plus multiple TTS/ASR/embedding vendors
  • Can run in pure voice mode with no local avatar GPU required
  • Built-in RTP metric to diagnose and tune realtime avatar streaming performance

Related

  • Project
    ddxfish/sapphire

    A self-hosted agentic framework for creating customizable AI personas with voice interaction, long-term memory, and autonomous system control.

  • Project
    datascale-ai/opentalking

    An open-source orchestration framework for real-time digital-human conversations, integrating LLMs, TTS, STT, and video rendering via WebRTC.

  • Project
    AlphaAvatar/AlphaAvatar

    AlphaAvatar is a self-hostable framework for building proactive, multimodal personal AI assistants that can remember, plan, and execute tasks using a plugin-based architecture.

  • Project
    TEN-framework/ten-framework

    An open-source framework for building real-time multimodal conversational AI agents with low-latency audio, text, and visual capabilities.

  • Project
    plurai-ai/intellagent

    A multi-agent framework that stress-tests conversational AI agents by simulating thousands of realistic edge-case interactions to uncover failure points and optimize performance.