code-video-model/CodeVideoModel

What it solves

Code Video Model addresses the challenge of generating high-quality, controllable video by using 3D scene code as a structural guide. Instead of relying solely on text-to-video models, which can struggle with spatial consistency and precise camera movements, this project uses Three.js code to define the scene's geometry and motion, ensuring the final video output is structurally accurate.

How it works

The process involves three main stages. First, it uses an LLM to generate Three.js code from a text prompt, creating a "proxy" video that defines the scene structure. Second, it generates a high-quality reference image that captures the desired visual style and composition. Finally, it uses the MiniMax-H3 model to synthesize the final video, combining the structural guidance from the proxy video with the visual details from the reference image.

Who it’s for

This project is for researchers and developers working on controllable video generation, 3D-to-video pipelines, and AI-driven scene synthesis who need a framework to bridge the gap between programmatic 3D environments and high-fidelity generative video models.

Highlights

• Code-Guided Generation: Uses Three.js code as a structural proxy to control camera paths and object placement.
• Multi-Stage Pipeline: Integrates LLM-based code generation, image-based style referencing, and video synthesis.
• Interactive Viewer: Includes a browser-based tool to inspect, scrub, and visualize the generated 3D scenes.
• Benchmark Included: Provides CodeVideoBench, a dataset of 22 cases across eight applications to evaluate structure, style, and visual quality.