NVIDIA-AI-Blueprints/video-search-and-summarization

NVIDIA AI Blueprint – Video Search & Summarization (VSS)

What it is – A reference implementation that shows how to build GPU‑accelerated video‑AI agents capable of searching, analysing, summarising and reasoning over live or recorded video using natural‑language queries. It stitches together vision‑language models (VLMs), large language models (LLMs), Retrieval‑Augmented Generation (RAG) and NVIDIA NIM micro‑services.

Key capabilities

• Natural‑language video search – index video streams with dense embeddings and retrieve relevant clips via text queries.
• Long‑video summarisation – chunk hours‑long footage, generate dense captions, and aggregate them into concise summaries.
• Visual Q&A & report generation – ask questions about a clip; a VLM produces answers and formatted reports.
• Real‑time alert verification – object‑detection & tracking pipelines raise alerts; a VLM validates them to cut false positives.
• Modular micro‑service stack – separate services for real‑time feature extraction, downstream analytics, and agent‑orchestrated offline tools.

How it’s organised

Layer	Role
Real‑time video intelligence	Extracts visual features, embeddings, and metadata from streams; publishes to a message broker.
Downstream analytics	Enriches metadata (behaviour analytics, incident detection) and stores results in Elasticsearch/Redis.
Agent & offline processing	Orchestrates tools via the Model Context Protocol (MCP) to perform search, Q&A, summarisation, and clip retrieval.

Main components

• NIM micro‑services – pre‑built NVIDIA models such as Cosmos3 Nano Reasoner and Nemotron‑Nano‑9B‑v2.
• services/agent/ – Python agent code (tools, APIs, embeddings, evaluators).
• services/ui/ – Next.js front‑end for interacting with the agent.
• services/analytics/ – Python and Node.js pipelines that turn raw video analytics into searchable incidents.
• deploy/ – Docker‑Compose files, Helm charts, and Brev launch scripts for quick deployment on cloud or on‑prem hardware.
• tools/ – Consumers for Redis/Kafka streams, logstash plugins, and utilities for dataset generation and calibration.
• skills/ – Agentskills‑compatible skill packages that let other agents invoke VSS functionality.

Who it’s for

• Video analysts / IT engineers who need a ready‑to‑run solution for video analytics and reporting.
• GenAI / ML engineers who want to customise pipelines, swap models, or integrate VSS into larger vision‑agent systems.

Getting started

1. Prerequisites – NVIDIA AI Enterprise licence, API key for NIM models, compatible GPU (see NVIDIA docs for exact GPU list).
2. Quick‑start options
   • Launchable (Brev) deployment – run a notebook that spins up a 2×RTX PRO 6000 SE instance on AWS and deploys the whole stack automatically.
   • Docker‑Compose deployment – install Docker Engine (28.3.3 – < 29.5.0), NVIDIA Container Toolkit, and run the provided compose.yml on Ubuntu 22.04/24.04, DGX, or Jetson platforms.
3. Follow the step‑by‑step guide in the official docs: https://docs.nvidia.com/vss/latest/quickstart.html.

Documentation & demo

• Live demo: https://build.nvidia.com/nvidia/video-search-and-summarization
• Full docs: https://docs.nvidia.com/vss/latest/index.html
• Architecture diagram is included in the repo (assets/vss-architecture.png).

License – See the repository’s LICENSE file (standard NVIDIA open‑source terms).

---

All details are taken directly from the repository’s README; no additional features are inferred.