joinly-ai/joinly

What it solves

Joinly provides a way for AI agents to join, listen to, and actively participate in video calls on platforms like Google Meet, Zoom, and Microsoft Teams. It bridges the gap between LLMs and live meeting environments, allowing agents to interact via voice and chat in real-time.

How it works

It acts as connector middleware that uses an MCP (Model Context Protocol) server to provide a set of tools and resources to an AI agent. The system handles the browser-based connection to the meeting, manages speech-to-text (STT) for listening, and text-to-speech (TTS) for speaking. It is designed to be modular, allowing users to bring their own LLM (including local models via Ollama) and choose from various STT/TTS providers like Whisper, Deepgram, ElevenLabs, or Kokoro.

Who it’s for

Developers building AI agents that need to interact with humans in live video conferences, or teams wanting to automate meeting tasks such as live note-taking, real-time information retrieval, or updating external tools (like GitHub or Notion) during a call.

Highlights

• Cross-platform support: Works with Google Meet, Zoom, and Microsoft Teams via the browser.
• Real-time interaction: Supports voice and chat responses with built-in logic for handling interruptions and multi-speaker flows.
• MCP-powered: Provides specific tools for joining/leaving meetings, speaking, sending chat messages, and retrieving transcripts or video snapshots.
• Flexible infrastructure: Supports self-hosting via Docker, GPU acceleration for faster transcription/speech, and integration with various LLM and audio providers.