mexicat/pdoom-video

What it solves

This project provides a system for creating a generative, code-rendered music video with word-synced karaoke typography. It solves the problem of creating high-fidelity, deterministic visual animations that are perfectly synchronized with audio timing, allowing for identical results in both real-time browser previews and high-resolution offline exports (up to 4K60).

How it works

The system uses a combination of audio analysis tools and a TypeScript/three.js renderer:

1. Audio Analysis: Python tools (using Demucs for stem separation, Whisper and CTC for forced alignment, and other analysis scripts) generate timing data for lyrics, beats, and loudness envelopes.
2. Rendering Engine: A TypeScript-based engine using three.js and Vite renders frames as deterministic functions of song time. It includes post-processing effects like bloom, halation, and grain.
3. Export Pipeline: A headless Chrome instance (via Playwright) renders raw frames, which are sent over WebSockets to ffmpeg for encoding into a high-quality video file.
4. Motion Blur: To ensure smooth motion, the renderer can average multiple sub-frames per frame, with the number of samples automatically adjusting based on the speed of the motion.

Who it’s for

This is for digital artists, creative coders, and music video producers who want to create precise, code-driven animations synchronized to audio using a web-tech stack.

Highlights

• Deterministic Rendering: Ensures the browser preview and the final 4K export are identical.
• High-Fidelity Export: Supports 1080p60 and 4K60 resolution with advanced motion blur sampling.
• Automated Audio Sync: Uses AI-driven audio analysis (Whisper, Demucs) to generate precise word-level lyric timings.
• GPU-Accelerated: Leverages three.js for complex scenes and ray-marched environments.