Xiaoc7r/DOVideo-AI

What it solves

DoVideoAI is designed to handle the challenges of analyzing long-form video content, which is typically time-consuming, resource-intensive, and costly in terms of AI API calls. It transforms long videos into structured, searchable, and traceable knowledge, preventing the "hallucinations" common in AI by requiring every conclusion to be backed by a specific timestamped evidence from the original video.

How it works

The system uses a multi-layered approach to process video data:

1. Multimodal Context Extraction: It uses FFmpeg for audio slicing and scene change detection. It then runs ASR (Automatic Speech Recognition) and OCR (Optical Character Recognition) in parallel to create a unified VideoSegment containing speech, text, and keyframes with timestamps.
2. Hybrid Retrieval: The system generates summaries, keywords, and embeddings for 5-minute segments of the video. It uses a combination of keyword matching and Qdrant vector search to retrieve the most relevant evidence for a user's query.
3. Agentic Loop: A specialized Agent loop (Planner $\rightarrow$ Executor $\rightarrow$ Critic) decomposes user goals into tasks. The Critic ensures that the output is structured and supported by timestamped evidence; if not, it triggers a targeted re-search of the video content.
4. Reliable Pipeline: To handle large files and long processing times, it employs chunked uploads with breakpoint resume, asynchronous processing via RocketMQ, and rate-limiting to control costs.

Who it’s for

It is for users who need to extract structured insights from long videos—such as students creating study guides, reviewers looking for logical flaws, or creators extracting highlights—without having to watch the entire video manually.

Highlights

• Evidence-Based AI: Every conclusion is bound to a timestamp, ensuring traceability and preventing AI fabrication.
• Asynchronous Processing: Uses RocketMQ and Redis to ensure video analysis doesn't block the main request thread.
• Multimodal Fusion: Combines ASR and OCR data into a single searchable timeline.
• Specialized Analysis Modes: Offers dedicated modes for General analysis, Learning (outlines/quizzes), Review (logical gaps), and Creation (scripts/hooks).
• Fault Tolerance: Features checkpointing via MySQL and Redis to allow task recovery from the last successful stage.