TEN-framework/ten-framework

Open-source framework for conversational voice AI agents

What it solves

TEN provides a framework for building real-time, multimodal conversational AI agents. It addresses the complexity of creating low-latency assistants that can handle simultaneous audio, text, and visual inputs and outputs, such as voice assistants with lip-sync avatars or real-time transcription tools.

How it works

TEN acts as an orchestration layer that allows developers to chain together various extensions for speech-to-text (STT), large language models (LLM), and text-to-speech (TTS). It supports both RTC and WebSocket connections and can be integrated with hardware like the ESP32-S3. The framework includes a designer tool (TMAN Designer) for configuring agent properties and managing extensions visually.

Who it’s for

Developers building real-time AI agents, such as voice assistants, interactive avatars, or hardware-integrated AI communication tools, who need a low-latency multimodal pipeline.

Highlights

  • Multimodal Capabilities: Supports audio, text, and visual outputs, including integration with lip-sync avatar vendors (e.g., HeyGen, Tavus).
  • Low Latency: Designed specifically for real-time conversational AI.
  • Extensible Ecosystem: Includes specialized components for Voice Activity Detection (VAD) and Turn Detection to enable full-duplex dialogue.
  • Hardware Integration: Provides guides for running agents on development boards like the ESP32-S3 Korvo V3.

Related

  • Project
    GetStream/Vision-Agents

    A framework for building low-latency, multi-modal AI agents that can watch, listen, and understand video in real time using a combination of CV models and LLMs.

  • Project
    livekit/agents

    A framework for building real-time, programmable multi-modal AI agents that can see, hear, and speak, featuring integrated job scheduling and telephony support.

  • Project
    pipecat-ai/pipecat

    An open-source Python framework for building real-time voice and multimodal AI agents with support for multi-agent orchestration and low-latency transports.

  • Project
    evalstate/fast-agent

    A flexible framework for building multimodal AI agents and workflows with deep Model Context Protocol (MCP) integration and advanced orchestration patterns.

  • Project
    livekit/agents-js

    A Node.js framework for building real-time, multi-modal voice AI agents that can see, hear, and understand using a plugin-based architecture for STT, LLM, and TTS.