MugheesMehdi07/langvio

A powerful AI library that bridges Large Language Models (LLMs) with video processing frameworks like YOLO, enabling seamless integration of vision and language for advanced video understanding, object detection, and contextual analysis.

What it solves

Analyzing images and videos usually means writing detection or tracking code for each task. Langvio removes that step: you ask a question in plain English, such as "Count all red cars" or "Find people wearing yellow," and it returns an explanation and annotated media. It combines a language model for understanding the query with a vision model for detecting and tracking objects, so the same tool handles counting, attribute search, spatial questions, and video summaries.

How it works

Langvio runs a pipeline with two cooperating parts. A Qwen-VL language processor parses your query into targets, attributes, and task parameters, then later explains the detection results. A Qwen-VL vision processor grounds objects in images and in sampled video keyframes, normalizes bounding boxes, and removes duplicate detections. For videos, detections on keyframes are linked into tracks using labels, box overlap, and predicted motion; boxes are carried between keyframes with CPU optical flow, with interpolation as a fallback. Descriptive queries like "the person wearing glasses" are checked against detailed crops, including multiple views of each video track.

A query like "How many cars are in this parking lot?" flows through query parsing, object grounding (or keyframe grounding for video), track association and crop matching, then explanation generation. The result includes an annotated image or video, a natural-language explanation, and structured detection data with summaries.

Langvio runs by default over a hosted OpenAI-compatible API (OpenRouter) using an API key. If no key is set and you have installed the local extra, it falls back to loading Qwen2.5-VL weights in-process via HuggingFace. The language and vision processors resolve their backend independently. A legacy path using YOLO-World v2 with ByteTracker remains available for classic open-vocabulary detection.

Who it's for

Developers and researchers who want to ask questions about visual media without writing bespoke detection pipelines. It suits quick experiments via the CLI or Flask web app, and building into larger applications through the Python API. Anyone comfortable with Python 3.10-3.12 and either an OpenRouter API key or enough compute to run a local vision-language model.

Highlights

  • Natural-language queries: "Count the cars", "Find all red objects", "What objects are on the table?"
  • Image and video support, with keyframe-based tracking, optical flow, and interpolation fallback
  • Qwen-VL powers both query parsing/explanation and object grounding, with independently configurable processors
  • Hosted API by default (OpenRouter), local in-process inference as fallback when no key is set
  • Attribute matching for descriptive queries, using crops and multiple track views
  • Video summaries: estimated distinct objects, peak simultaneous counts, movement patterns, centerline crossings
  • Annotated output images/videos with bounding boxes and tracking trails, plus a Flask web interface
  • Extensible via YAML config; legacy YOLO-World + ByteTracker path still available

Related

  • Project
    OpenBMB/MiniCPM-o-Demo

    A demo system for MiniCPM-o 4.5 that enables real-time, full-duplex omnimodal interaction, allowing the AI to see, listen, and speak simultaneously.

  • Project
    OmniCustom-project/OmniCustom

    OmniCustom is a joint audio-video generation framework that creates synchronized videos preserving a specific visual identity and voice timbre from reference samples, guided by text prompts.

  • Project
    yuanze-lin/Olympus

    A universal task router for computer vision that converts a single natural-language instruction into a sequence of specialist model calls to produce images, videos, and 3D models.

  • Project
    zeroQiaoba/AffectGPT

    A research-focused framework for Open-Vocabulary Multimodal Emotion Recognition that uses generative MLLMs and reinforcement learning to understand the full spectrum of human emotions.