jd-opensource/JoyAI-VL-Interaction

What it solves

Most AI models are turn-based, meaning they only respond after being prompted. JoyAI-VL-Interaction solves this by creating a vision-language model that is "present" in a scene—it continuously watches a video stream and proactively decides when to speak, stay silent, or delegate a complex task to another agent, allowing for real-time, event-driven interaction without needing a user trigger.

How it works

The system is built around an 8B-scale vision-first model (JoyAI-VL-8B) that makes a decision every second based on the visual input. It uses a predictive video codec called AdaCodec to manage token budgets by spending fewer tokens on predictable frames and more on scene changes. The model was trained on over four million time-aligned interaction samples and refined using reinforcement learning. The full deployment stack includes five pluggable services: inference, a WebUI for streaming, ASR (speech recognition), TTS (speech synthesis), and a background agent for task delegation.

Who it’s for

This project is for developers and researchers building real-time AI assistants, monitoring systems, or interactive video applications that require proactive visual awareness and sub-second response times.

Highlights

• Real-time Presence: Capable of responding to visual events in under a second.
• Vision-triggered Proactivity: Can initiate conversation based on what it sees rather than waiting for a prompt.
• Agent Delegation: Can hand off difficult subtasks to background models or APIs while continuing to monitor the stream.
• Open Stack: Provides the model weights, training recipes, time-aligned interaction data, and a complete deployable system.
• Unified Capability: A single model that handles both real-time streaming interaction and offline video understanding.