Correr-Zhou/OmniShow

What it solves

OmniShow is designed for Human-Object Interaction Video Generation (HOIVG). It solves the problem of generating high-fidelity videos where humans interact with objects, while allowing users to control the output using multiple types of inputs (multimodal conditions) simultaneously, such as text, reference images of the person and object, audio, and pose sequences.

How it works

OmniShow uses a unified framework to handle various generation tasks (R2V, RA2V, RP2V, and RAP2V) through three key technical components:

1. Unified Channel-wise Conditioning: This method injects reference images and pose cues by concatenating them into the video tokens, using pseudo-frames and a reconstruction loss to keep semantic details intact.
2. Gated Local-Context Attention: This ensures audio-visual synchronization by packing audio features with context and using masked attention and adaptive gating to align video frames with audio segments.
3. Decoupled-Then-Joint Training: The model is trained in stages: first, specialized models for reference-to-video (R2V) and audio-to-video (A2V) are trained on separate datasets, then fused via weight interpolation and jointly fine-tuned.

Who it’s for

This project is for researchers and developers working on controllable video generation, digital avatars, and multimodal AI, specifically those interested in creating realistic human-object interactions.

Highlights

• All-in-one Model: The first model to unify text, reference images, audio, and pose conditioning for HOIVG.
• Flexible Task Support: Supports multiple configurations, including Reference+Audio+Pose-to-Video (RAP2V).
• Comprehensive Benchmark: Introduces HOIVG-Bench, a dedicated dataset for evaluating multimodal HOIVG.
• Versatile Applications: Capable of audio-driven avatars, object swapping, and video remixing.