andyhuo520/digital-human-koubo

Digital Human Koubo is an open-source project that turns one photo + one sentence into a finished talking-head video with a digital human presenter. Everything — script, lip-synced AI presenter, voice clean-up, captions, and pro-YouTuber-style packaging — is automated by an AI agent.

What it does

• Digital human from one photo — a fixed presenter (face, outfit, voice description) is stored in persona.json, so every video uses the same person. You can composite her into any scene (studio, shop, classroom, street) using GPT Image, then make her talk.
• Lip-synced talking clips — the script is split into ≤15-second segments and generated with Grok Video 1.5 via Renoise MCP. Other models (Wan 3.0, OmniHuman) are documented.
• Cost gate — the agent always quotes segments · seconds · credits · balance first and waits for your approval. It generates segment 1 for you to check before batching the rest.
• Quality control — every segment is transcribed with whisper and compared with the script; 4 frames per segment are inspected (Grok sometimes burns subtitles into the picture; those get re-shot).
• Voice unification — spectral matching, de-essing, shared reverb, and loudness normalization make 30 separately-generated clips sound like one recording.
• 5 editing styles — the agent diagnoses your content, recommends 2 styles with preview images, and asks you everything in one go (style, ratio, length, caption intensity, presenter).
• 48 packaging components — measured frame-by-frame from 25 top creators (MrBeast, MKBHD, 何同学, etc.). Each style has a default recipe; you can add/remove techniques by just asking.
• Text behind the presenter — big words/numbers between the background and the person (macOS Vision person segmentation → transparent video → 3-layer composite).
• Multi-language — English, Japanese, Korean, French, Spanish, Arabic tested (whisper back-check 94–100%).
• Works with your agent — a standard SKILL.md Agent Skill plus scripts; any agent that can load skills, connect a remote MCP server, and run local commands can drive it.

How it works

1. Setup check (Renoise MCP / ffmpeg / whisper — skip what you have).
2. Diagnose content → recommend 2 of 5 styles → ask once → brief.json.
3. Script (≤15 s per segment, ~5.4 Chinese chars/s).
4. Quote credits → wait for "yes" → generate segment 1 → you check → batch.
5. Range-download (CDN-safe) → whisper back-check → frame check.
6. Voice unification (spectral match + de-ess + shared room).
7. Word-level timeline (tl.json).
8. Edit in Remotion: style template + default packaging (+ your requests).
9. Render → check mid-transition frames → loudness (-14 / -16 LUFS).

Supported agents

• Claude Code — tested, built with it.
• Codex — expected to work.
• OpenClaw, WorkBuddy, 豆包, 千问办公, 纳米Work — should work if they support the three requirements (skills, MCP, local commands).
• Claude desktop / claude.ai — can plan, quote, and generate clips, but local editing needs a shell agent.

Installation

1. Clone the repo.
2. Install the two skills (digital-human-koubo and finance-explainer) into your agent (Claude Code, Codex, or zip for others).
3. Connect Renoise MCP (browser login, no API key).
4. Install local tools: ffmpeg, Python 3 + numpy + Pillow, Node.js 18+ (Remotion), whisper.cpp.
5. Generate sound effects & placeholder music (no third-party audio shipped).
6. (Optional) Xiaomi MiMo TTS for designed/cloned voices — only clone your own or with permission.

Quick start

Talk to your agent in plain language, e.g., "Use my digital human skill to make a 30-second livestream ad: WorkBuddy course, 10 lessons, ¥99 (was ¥999)." The agent runs the pipeline automatically.

Make your own digital human

1. Pick a frontal photo (even light, hair off face, mouth closed) — only your own or with permission.
2. Generate a 4-view sheet (front, sides, back) with GPT Image 2.5.
3. Compare models (Grok Video 1.5 vs Wan 3.0) and pick one.
4. Upload the reference to Renoise and put its material:<id> into persona.json.

Five editing styles

Style	Ratio	Length	Best for
A Finance/tech explainer	16:9	3–20 min	events, data, "why…"
B Influencer talking head	9:16	30–120 s	product picks, sales, opinions
C Launch breakdown	16:9	30–90 s	new products/models
D Step-by-step tutorial	16:9	3–8 min	"how to…", tool setup
E Multi-host · multi-language · multi-scene	16:9	3–7 min	capability demos, going global

Style B has three caption intensities: restrained / standard / lively.

Demos

Six full videos are linked in the README (livestream ad, product recommendation, text-behind, tutorial, launch breakdown, finance explainer) plus a packaging gallery. All presenters are AI digital humans; the male presenter is the author's own digital twin.

License: MIT. All demos were produced with this repo; total cost for the 27-second ad was 482 Renoise credits.