HKUDS/ViMax

📽️ ViMax – Agentic End‑to‑End Video Generation

What it is – ViMax is an open‑source framework that lets you go from a plain text idea (or a full screenplay) to a finished video without stitching together separate tools. It does this by running a loop of LLM‑driven agents that write scripts, design storyboards, generate consistent character images, and finally call an AI video generator. The whole pipeline can be driven from a terminal UI or a browser‑based Web UI.

---

✨ Core capabilities

Feature	What you get
Idea2Video	Feed a short concept; ViMax creates a story outline, characters, script, shot list, storyboard and renders a short clip.
Script2Video	Provide a screenplay; the system plans shots, keeps visual continuity, and produces a multi‑scene video.
Novel2Video	Turn a longer piece of fiction into an episodic video, handling narrative compression and character tracking.
AutoCameo	Upload a photo of a person or pet and have that subject appear consistently throughout the generated story.
Agent Loop + TUI	Interactive, chat‑style planning where you can revise ideas, resume sessions, and monitor render status from the terminal.
Web UI	Browser workspace with named projects, artifact & storyboard previews, render checkpoints, file uploads and provider configuration (dark mode available).
Parallelised generation	Shots and media assets are generated concurrently to speed up multi‑shot productions.
Provider‑agnostic	Plug‑in support for any LLM, image model, or video model that offers an HTTP API (OpenAI, OpenRouter, Google Gemini, Seedance, etc.).

---

🛠️ Quick‑start (Linux / Windows)

# 1️⃣ Clone the repo
git clone https://github.com/HKUDS/ViMax.git && cd ViMax

# 2️⃣ Install the Python environment with uv (or pip if you prefer)
uv sync   # creates a virtualenv and installs dependencies

Run the terminal UI

# copy the example config and fill in your API keys
cp configs/agent.example.yaml configs/agent.local.yaml
# edit the yaml – set LLM, image and video model/provider details
vim configs/agent.local.yaml

# start the interactive TUI
vimax tui          # new session
vimax tui new      # or resume an existing one

Run the browser UI

cd web
npm install        # one‑time front‑end install (Node 18+ required)
pm run dev        # starts the dev server on http://127.0.0.1:4173

If the backend runs on a remote machine, forward the port with SSH as described in the README.

---

📂 Example workflows

• Idea2Video – edit main_idea2video.py with a short prompt and optional style/requirements, then run it. The script reads configs/idea2video.yaml for the LLM, image and video endpoints.
• Script2Video – edit main_script2video.py with a full screenplay and call the script. Configuration lives in configs/script2video.yaml. Both scripts create a working directory (.working_dir/...) that stores generated text, images, storyboards and the final video.

---

📦 What’s inside the repo?

Directory / file	Purpose
configs/	Example YAML files for the three model providers (LLM, image, video).
web/	React‑based front‑end; npm run dev serves the UI.
tools/	Python classes that wrap specific image/video APIs (e.g., ImageGeneratorNanobananaGoogleAPI).
main_idea2video.py / main_script2video.py	Minimal entry‑points that demonstrate the two main pipelines.
README.md	This documentation, plus badges, news, demos and quick‑start instructions.

---

🚀 Recent highlights (as of the README)

• v1.2.0 – Web UI with named projects, artifact previews, render checkpoints and dark mode.
• Added OpenRouter GPT‑Image‑2‑Image and Seedance 2.0 fast video generation.
• Integrated Agent Loop + TUI for interactive planning and session reuse.
• Released a technical report and a Novel2Video workflow.

---

🎯 Who might use ViMax?

• Indie creators who want to prototype animated shorts without hiring a full production crew.
• Educators looking for quick, illustrated story videos for lessons.
• Researchers exploring multimodal LLM pipelines (text → storyboard → video) and needing a modular, extensible codebase.
• Developers who want to plug in their own image or video generation models.

---

📜 License

MIT – you can freely use, modify and redistribute the code.

---

📚 Further reading

• ArXiv paper – 2606.07649 (linked in the badge).
• YouTube channel – “AI‑Creator‑is‑here” for demo videos and tutorials.

---

✅ TL;DR

ViMax bundles LLM‑driven planning, consistent image generation, and AI video synthesis into a single, agent‑centric workflow. With a terminal UI, a modern Web UI, and plug‑in support for any LLM/image/video provider, it lets you type a story idea and receive a finished video—making AI‑generated video production far more accessible.