livekit/agents
A framework for building realtime voice AI agents 🤖🎙️📹
What it solves
LiveKit Agents provides a framework for building real-time, programmable AI participants that run on servers. It simplifies the creation of multi-modal voice agents capable of seeing, hearing, and understanding, while handling the complexities of server-side execution and user connectivity.
How it works
The framework uses an AgentServer to coordinate job scheduling and launch agents for user sessions. Developers define an Agent with specific instructions and tools, and an AgentSession manages the interaction. The system allows developers to mix and match various Speech-to-Text (STT), Large Language Model (LLM), and Text-to-Speech (TTS) providers through a flexible plugin system, or use a unified API for hosted models.
Who it’s for
Developers building real-time AI voice assistants, multi-modal agents, and telephony-integrated AI applications that require low-latency interaction and server-side orchestration.
Highlights
- Flexible Integrations: Mix and match STT, LLM, and TTS providers to suit specific use cases.
- Telephony Integration: Seamlessly connects agents to phone calls via LiveKit's SIP stack.
- Semantic Turn Detection: Uses a transformer model to reduce interruptions by detecting when a user has finished speaking.
- MCP Support: Native integration with Model Context Protocol (MCP) servers to add tools with minimal code.
- Built-in Testing: Includes a test framework with LLM-based judges to ensure agent performance.
- Multi-agent Handoff: Supports transitioning interactions between different specialized agents.