TEN-framework/ten-framework

What it solves

TEN provides a framework for building real-time, multimodal conversational AI agents. It addresses the complexity of creating low-latency assistants that can handle simultaneous audio, text, and visual inputs and outputs, such as voice assistants with lip-syncing avatars or real-time transcription tools.

How it works

TEN operates as an extensible framework where developers can chain together various extensions for speech-to-text (STT), large language models (LLM), and text-to-speech (TTS). It supports both RTC and WebSocket connections and can be integrated with hardware like the ESP32-S3. The system includes a "TMAN Designer" for visual configuration of agent properties and API keys.

Who it’s for

It is designed for developers building interactive AI applications, ranging from simple voice assistants and transcription tools to complex multimodal experiences like real-time speaker diarization or animated avatars.

Highlights

• Multimodal Support: Handles audio, text, and visual data for real-time interaction.
• Low Latency: Optimized for high-quality, real-time conversational experiences.
• Extensible Ecosystem: Includes dedicated tools for Voice Activity Detection (VAD) and Turn Detection to enable full-duplex dialogue.
• Hardware Integration: Supports deployment on development boards like the ESP32-S3 Korvo V3.
• Flexible Deployment: Can be hosted via Docker or split between a containerized backend and a hosted frontend (e.g., Vercel/Netlify).