NVIDIA-AI-Blueprints/video-search-and-summarization

NVIDIA AI Blueprint for video search and summarization (VSS) is a GPU-accelerated reference architecture for building video analytics agents with real-time verified alerts, visual Q&A, and automated reporting. The VSS Blueprint uses vision language models (VLMs) such as NVIDIA Cosmos, LLMs such as NVIDIA Nemotron, RAG, and NVIDIA NIMs.

What it solves

This project addresses the difficulty of analyzing and interacting with massive volumes of both stored and streaming video data. It enables the creation of vision AI agents that can perform natural-language search, summarize long recordings, and verify alerts in real-time to improve operational efficiency in environments like smart spaces and warehouses.

How it works

The system uses a layered architecture combining vision microservices, Vision Language Models (VLMs), and Large Language Models (LLMs):

  • Real-time Video Intelligence: Extracts visual features, semantic embeddings, and contextual understanding from live streams, publishing them to a message broker.
  • Downstream Analytics: Transforms raw detections into actionable insights, such as trajectories, incidents, and verified alerts.
  • Agentic Processing: A top-level agent uses the Model Context Protocol (MCP) to access video analytics data and tools for Q&A, semantic search, and clip retrieval.

Who it’s for

  • Video Analysts and IT Engineers: Those who need a ready-to-deploy solution for processing and summarizing video data via 1-click deployments.
  • GenAI Developers and ML Engineers: Experts looking to customize pipelines for specific datasets or fine-tune LLMs for specialized use cases.

Highlights

  • Diverse Agent Workflows: Supports Q&A report generation, real-time anomaly detection, and long video summarization through chunking and dense captioning.
  • VLM-based Verification: Uses VLMs to verify alerts generated by perception models to reduce false positives.
  • Semantic Search: Enables natural language searching across video archives using video embeddings.
  • NVIDIA NIM Integration: Leverages optimized microservices like Cosmos3 Nano Reasoner and Nemotron-Nano-9B-v2.