HKUDS/VideoAgent

What it solves

VideoAgent is an all-in-one framework designed to automate the complex process of understanding, editing, and generating creative video content. It removes the need for technical expertise or complex interfaces, allowing users to create professional-quality videos—such as movie edits, meme videos, and music videos—using only natural language prompts.

How it works

The system operates through a multi-modal agentic framework consisting of three core innovations:

• Intent Analysis: It decomposes user instructions into explicit and implicit sub-intents and maps them to specific agent capabilities to ensure precise task execution.
• Autonomous Tool Use & Planning: Using a graph-powered framework, it translates intents into executable workflows. It employs adaptive feedback loops and two-step self-evaluation to refine and self-correct the planning process.
• Multi-Modal Understanding: A Storyboard Agent analyzes video material banks and transforms user input into fine-grained, semantically aligned visual queries to retrieve the most relevant video segments.

Who it’s for

This tool is for content creators, AI researchers, and anyone who wants to produce high-quality video content (like commentary videos, news overviews, or cross-cultural comedy) without needing professional editing skills.

Highlights

• Conversational AI Interface: Full video creation and interaction via natural dialogue.
• Diverse Creative Outputs: Supports beat-synced edits, storytelling videos, meme remaking, and song remixes.
• Self-Improving Workflows: An adaptive reflection mechanism that increases workflow composition success rates.
• Integrated Toolset: Leverages multiple specialized models for TTS, SVC, and video retrieval (e.g., CosyVoice, Fish Speech, Whisper, ImageBind).