livekit/agents

LiveKit Agents – Python framework for real‑time AI voice agents

What it is – A Python SDK that lets you build server‑side agents that can listen, speak, and even see in real time. An agent is essentially an LLM‑driven application (e.g., a voice assistant) that runs inside the LiveKit ecosystem and talks to users through WebRTC or phone calls.

Core ideas

• Agent – a piece of code that wraps an LLM, optional tools (function calls), and a set of instructions.
• AgentSession – manages the media pipeline (speech‑to‑text, turn‑detection, text‑to‑speech) and the conversation state for a single user or room.
• AgentServer – the process that schedules sessions, launches agents, and connects them to LiveKit rooms.
• Plugins – adapters for popular model providers (OpenAI, Deepgram, Cartesia, Google, etc.) that expose a unified inference.* API.

Key features

• Mix‑and‑match any STT, LLM, and TTS service.
• Built‑in job dispatching so many users can be served concurrently.
• Full WebRTC client support (browser, mobile, Unity, etc.) and telephony via LiveKit’s SIP stack.
• Turn‑detection transformer to know when a speaker has finished.
• One‑line MCP integration for external tool servers.
• Native test framework with “judges” to assert LLM behaviour.
• Open‑source and can run entirely on self‑hosted LiveKit servers.

Installation

pip install "livekit-agents[openai,deepgram,cartesia]"

The extra brackets pull in optional plugins for the most common model providers.

Quick start (voice assistant)

from livekit.agents import Agent, AgentServer, AgentSession, JobContext, RunContext, cli, function_tool, inference

@function_tool
async def lookup_weather(context: RunContext, location: str):
    return {"weather": "sunny", "temperature": 70}

server = AgentServer()

@server.rtc_session()
async def entrypoint(ctx: JobContext):
    session = AgentSession(
        vad=inference.VAD(),
        stt=inference.STT("deepgram/nova-3", language="multi"),
        llm=inference.LLM("google/gemma-4-31b-it"),
        tts=inference.TTS("cartesia/sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"),
    )
    agent = Agent(instructions="You are a friendly voice assistant.", tools=[lookup_weather])
    await session.start(agent=agent, room=ctx.room)
    await session.generate_reply(instructions="greet the user and ask about their day")

if __name__ == "__main__":
    cli.run_app(server)

Set LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET so the process can join a LiveKit room.

More advanced patterns

• Multi‑agent handoff – one agent gathers information, then hands the conversation to a specialized agent (e.g., a storyteller). Hand‑off is done by returning a new Agent instance from a tool call.
• Testing – the SDK ships with a pytest‑compatible test harness that can assert that a given user utterance triggers the expected function call or LLM response.
• MCP tools – connect to a LiveKit Docs MCP server to let agents query up‑to‑date documentation or code search.

Running & development

Mode	Command	What it does
Console (quick test)	python myagent.py console	Runs locally with microphone/audio I/O, no server needed
Development	python myagent.py dev	Starts AgentServer with hot‑reloading; connects to LiveKit Cloud or self‑hosted server
Production	python myagent.py start	Runs with production‑grade optimisations

Resources

• Full docs: https://docs.livekit.io/agents/
• Example repo: https://github.com/livekit-examples/python-agents-examples
• Community Slack, Twitter, and Discord links are in the README badges.

License – Apache‑2.0 for the framework; the turn‑detection model has its own LiveKit Model License.

---

LiveKit Agents lets developers turn any LLM into a real‑time voice (or video) participant that can be called from browsers, mobile apps, or phones, all powered by LiveKit’s WebRTC infrastructure.