livekit/agents

A framework for building realtime voice AI agents 🤖🎙️📹

What it solves

LiveKit Agents provides a framework for building real-time, programmable participants that run on servers. It simplifies the creation of multi-modal voice agents capable of seeing, hearing, and understanding, while handling the complex infrastructure required for low-latency interactions.

How it works

The framework uses an AgentServer to coordinate job scheduling and launch agents for user sessions. Developers define an Agent with specific instructions and tools, and an AgentSession manages the interaction. The system allows developers to mix and match different providers for Speech-to-Text (STT), Large Language Models (LLMs), and Text-to-Speech (TTS), or use a unified API for these services. It also integrates with WebRTC clients and telephony stacks (SIP) for voice calls.

Who it’s for

Developers building conversational AI applications, such as voice assistants, outbound callers, and AI avatars, who need a scalable, server-side framework to manage real-time multi-modal interactions.

Highlights

  • Flexible Integrations: Support for various STT, LLM, and TTS providers.
  • Semantic Turn Detection: Uses a transformer model to detect when a user has finished speaking to reduce interruptions.
  • MCP Support: Native integration with Model Context Protocol (MCP) servers to add tools with minimal code.
  • Telephony Integration: Ability to make and receive phone calls via SIP.
  • Built-in Testing: A native test framework with judges to validate non-deterministic LLM behavior.
  • Multi-agent Handoff: Support for transferring a user session between different specialized agents.

Related

  • Project
    livekit/agents-js

    A Node.js framework for building real-time, multi-modal voice AI agents that can see, hear, and understand using a plugin-based architecture for STT, LLM, and TTS.

  • Project
    livekit-examples/python-agents-examples

    A collection of runnable Python examples and full-stack applications for building voice, video, and telephony AI agents using the LiveKit Agents framework.

  • Project
    GetStream/Vision-Agents

    A framework for building low-latency, multi-modal AI agents that can watch, listen, and understand video in real time using a combination of CV models and LLMs.

  • Project
    pipecat-ai/pipecat

    Pipecat is an open‑source Python framework for building real‑time voice and multimodal AI agents. It offers plug‑and‑play support for dozens of speech‑to‑text, large‑language‑model, text‑to‑speech, and speech‑to‑speech services, composable pipelines, multi‑agent orchestration, and client SDKs for web, mobile, and embedded platforms. The Pipecat CLI can scaffold a working bot in minutes, and additional tools (Whisker debugger, Tail dashboard, Voice UI Kit) help with development and deployment.

  • Project
    evalstate/fast-agent

    A flexible framework for building multimodal AI agents and workflows with deep Model Context Protocol (MCP) integration and advanced orchestration patterns.