livekit/agents-js

What it solves

It provides a framework for building real-time, programmable AI agents that act as participants in a server-side environment. It specifically addresses the complexity of creating multi-modal voice agents that can see, hear, and understand in real-time using WebRTC.

How it works

The framework uses a worker-based orchestration system where a worker process connects to a LiveKit server and waits for jobs. When a user joins a room, the worker instantiates an agent as a participant. These agents are built by combining plugins for Speech-to-Text (STT), Large Language Models (LLM), and Text-to-Speech (TTS), along with Voice Activity Detection (VAD) and semantic turn detection to manage natural conversation flow.

Who it’s for

Developers building conversational AI applications, voice assistants, and multi-modal agents that require low-latency, real-time interaction across various platforms.

Highlights

• Plugin Ecosystem: Extensive support for providers like OpenAI, Google, Deepgram, ElevenLabs, and Mistral AI.
• Semantic Turn Detection: Uses transformer models to reduce interruptions by detecting when a user has finished speaking.
• MCP Support: Native integration with Model Context Protocol (MCP) servers to add tools to agents with minimal code.
• Multi-agent Handoff: Built-in capability to transfer a user session from one specialized agent to another.
• Open Source: Compatible with the open-source LiveKit server for full self-hosting.