GetStream/Vision-Agents
Open Vision Agents by Stream. Build voice and vision agents quickly with any model or video provider. Uses Stream's edge network for ultra-low latency.
What it solves
Vision Agents provides a framework for building low-latency, multi-modal AI agents that can see, hear, and speak in real-time. It addresses the difficulty of combining real-time video streaming (WebRTC) with large language models (LLMs) and specialized computer vision models (like YOLO) to create interactive, visual AI experiences without significant lag.
How it works
The project uses a pluggable architecture that allows developers to stream video directly to model providers. It combines a video processing pipeline (supporting YOLO, Roboflow, and custom PyTorch/ONNX models) with native API integrations for LLMs like Gemini, OpenAI, and Claude. It also integrates with various speech-to-text (STT) and text-to-speech (TTS) providers to handle natural conversation flow, including turn detection and voice-to-voice interactions.
Who it’s for
Developers building real-time AI applications such as AI sports coaching, drone fire detection, physical therapy assistants, virtual try-ons, and automated security monitoring.
Highlights
- Ultra-low latency: Maintains audio/video latency under 30ms using Stream's edge network.
- Multi-modal integration: Seamlessly combines specialized CV models with general-purpose LLMs.
- Extensible pipeline: Pluggable processors for frame-by-frame understanding.
- Broad ecosystem: Native SDKs for React, Android, iOS, Flutter, and Unity.
- Production-ready: Includes built-in HTTP server, Prometheus metrics, and Kubernetes deployment support.
- Advanced capabilities: Supports RAG, tool calling via MCP, and bidirectional phone integration via Twilio or Telnyx.