Mr-funny/hbg-classical-poem-silk-video

What it is

hbg‑classical‑poem‑silk‑video is an Agent Skill (a plug‑in for AI coding agents such as Codex or Claude Code) that turns a Chinese classical poem into a vertical‑format (1080 × 1920) “silk‑video” – a short, high‑quality Chinese‑painting‑style animation. The skill parses the poem, splits it into scenes, generates static Chinese‑painting frames with an internal ImageGen model, feeds those frames to a Gemini‑based image‑to‑video (I2V) Docker container, adds motion cues (water ripples, birds, horse steps, etc.), overlays brush‑stroke subtitles, mixes the original model‑generated ambient sound with optional background music, and finally produces a QA‑checked MP4.

---

How to get it

The repository ships an install.sh script that can be run directly from the command line. The script detects the user’s agent (Codex or Claude Code) and copies the skill files into the agent’s global skills directory (default ~/.codex/skills/classical-poem-silk-video). Example one‑liner:

curl -fsSL https://raw.githubusercontent.com/Mr-funny/hbg-classical-poem-silk-video/main/install.sh | sh

You can also clone the repo manually and copy the skill/classical-poem-silk-video folder into the appropriate location.

---

Running the skill

After installation, talk to your agent with natural language, e.g.:

使用 $classical-poem-silk-video 制作《枫桥夜泊》。
四句诗一行一景，画面色调跟随诗意变化；镜头固定，
让月光、水纹、乌鸦、霜雾和渔火在画中自然运动。
保留 AI 环境原声，毛笔字逐字出现，场景之间自然交叉溶解。

The agent will:

1. Parse the poem (identify era, season, imagery, emotional arc).
2. Storyboard – one scene per line for short poems, two lines per scene for longer ones.
3. Select a Chinese‑painting style (ink‑wash, gong‑bi, qing‑lu, horse‑riding, etc.) based on the poem’s mood.
4. Generate static frames with the built‑in ImageGen, reserving a safe zone for vertical brush‑stroke subtitles.
5. Call the Gemini I2V Docker container (gemini-flow-suite) to animate the frames, using carefully crafted motion prompts that lock the camera and only move the intended objects.
6. Add subtitles in the Ma Shan Zheng brush font (right‑column first, then left‑column).
7. Mix audio – keep the model‑generated ambient sound and optionally overlay a background‑music track.
8. Cross‑fade transitions (both video and audio) and run a final QA script that checks for black frames, audio gaps, proper frame count, and visual consistency (e.g., no extra birds, malformed horse legs).

The result is a 1080 × 1920, 30 fps H.264 + AAC MP4 ready for sharing.

---

Key technical pieces

Component	What it does
Poem parser	Extracts era, location, season, imagery, actions, and emotional curve.
Storyboard engine	Groups lines (≤4 lines → one‑line‑per‑scene; longer → two‑lines‑per‑scene).
Style selector	Switches among ink‑wash, gong‑bi, green‑mountain, horse‑riding branches while keeping a unified Chinese‑painting DNA.
Static frame generator	Calls the agent’s built‑in ImageGen model; reserves a vertical subtitle safe‑zone.
Docker I2V	Runs the Gemini Flow Suite container (gemini-flow-suite) to turn frames into short video clips with controlled local motion.
Motion prompt template	Four‑section prompt (static anchor, local motion area, stable ending, anti‑hallucination rule) that prevents unwanted “rewriting” of the painting.
Subtitle renderer	Uses the SIL‑OFL Ma Shan Zheng brush font, vertical two‑column layout, character‑by‑character reveal.
Audio mixer	Preserves the model‑generated ambient track, optionally mixes a user‑provided BGM, and aligns it with cross‑fade transitions.
Final QA script	Uses ffprobe, ffmpeg filters (blackdetect, silencedetect, volumedetect) and optional timestamp TSV to verify motion consistency, frame integrity, and encoding specs.

---

Who might use it

• Content creators who want AI‑generated, culturally authentic short videos for social media (Weibo, Douyin, Bilibili).
• Educators looking for animated illustrations of classical poetry.
• Developers of AI agents who need a ready‑made skill to showcase multimodal generation (text → image → video → audio).
• Researchers interested in prompt engineering for controlled image‑to‑video generation.

---

License & compliance

• All scripts, the skill definition, and documentation are released under the MIT License.
• The bundled Ma Shan Zheng font is under the SIL Open Font License 1.1.
• The skill explicitly avoids handling any secret tokens, cookies, or user‑provided media; it only operates on the poem text and optional BGM supplied by the user.

---

Quick start checklist

1. Install the skill with the one‑liner above.
2. Ensure Docker is running and the gemini-flow-suite image is available (follow its README).
3. Run skill/classical-poem-silk-video/scripts/check_prerequisites.sh to verify the environment.
4. Ask your agent to generate a video for a poem of your choice.
5. When the MP4 is produced, optionally run final_media_qa.sh to confirm quality.

---

Example output

The repository includes a full‑length example for the poem 《钱塘湖春行》 (8 lines → 4 scenes). The final video (≈32 s, 1080 × 1920) is available in the Releases page and is discussed in docs/article.md.