Kevin-thu/StoryMem

What it solves

StoryMem addresses the challenge of creating long, multi-shot narrative videos that maintain consistent characters and cinematic quality across different scenes. Traditional video generation often struggles with "character drift" or visual inconsistency when generating a sequence of shots based on a story script.

How it works

The system uses a memory-conditioned single-shot video diffusion model to generate videos shot-by-shot. It starts by generating an initial shot (using a text-to-video model) to establish a baseline memory. For subsequent shots, it uses a specialized Memory-to-Video (M2V) LoRA fine-tuned on the Wan2.2 base model. The process involves:

• Memory Management: The model automatically extracts keyframes from generated shots and updates a memory buffer to maintain visual coherence.
• Conditioning Options: It supports different conditioning modes, including M2V (memory only), MI2V (memory plus the first-frame image of the next shot), and MM2V (memory plus the first five motion frames).
• Script-Driven Generation: It follows a structured JSON story script that defines scene cuts and specific prompts for each shot.

Who it’s for

This tool is designed for AI researchers, digital storytellers, and creators who need to generate cohesive, minute-long narrative videos with stable character identities and environments.

Highlights

• Multi-shot Consistency: Maintains character and visual coherence across multiple shots using a memory mechanism.
• Flexible Conditioning: Offers multiple modes (M2V, MI2V, MM2V) to handle both scene cuts and smooth transitions between adjacent shots.
• ST-Bench: Includes a new evaluation benchmark consisting of 30 long story scripts with 300 detailed video prompts.
• Wan2.2 Integration: Built upon the Wan2.2 MoE video diffusion architecture.