Correr-Zhou/OmniShow

[ICML 2026] ByteDance's All-in-One Video Generation Model for Human-Object Interaction Video Generation

What it solves

OmniShow is designed to solve the challenge of generating high-fidelity videos of humans interacting with objects (Human-Object Interaction Video Generation or HOIVG) using multiple types of control signals. Previous models often struggled to unify different modalities—such as text, images, audio, and poses—into a single framework that maintains visual consistency and synchronization.

How it works

OmniShow uses a unified framework based on Wan models to process various inputs:

  • Unified Channel-wise Conditioning: It injects reference images and pose cues by concatenating them as channels, using pseudo-frames and a reconstruction loss to keep semantic details intact.
  • Gated Local-Context Attention: This mechanism aligns audio features with video frames using masked attention and adaptive gating, ensuring the audio and visual elements are synchronized.
  • Decoupled-Then-Joint Training: The model is first trained on specialized sub-tasks (like Reference-to-Video and Audio-to-Video) and then fused via weight interpolation and joint fine-tuning to combine all capabilities into one model.

Who it’s for

This project is for researchers and developers working on multimodal video generation, specifically those focusing on human-centric content, audio-driven avatars, and controllable video synthesis.

Highlights

  • All-in-one Model: Supports text, reference images, audio, and pose conditioning simultaneously.
  • Flexible Task Coverage: Handles various generation modes including R2V (Reference), RA2V (Reference + Audio), RP2V (Reference + Pose), and RAP2V (Reference + Audio + Pose).
  • Comprehensive Benchmark: Introduces HOIVG-Bench, a dedicated dataset and evaluation suite for multimodal HOIVG.
  • Versatile Applications: Capable of object swapping, video remixing, and creating audio-driven avatars.

Related

  • Project
    OmniCustom-project/OmniCustom

    OmniCustom is a joint audio-video generation framework that creates synchronized videos preserving a specific visual identity and voice timbre based on reference inputs and text prompts.

  • Project
    Ma-Zhuang/OmniNWM

    OmniNWM is a research codebase that builds a panoramic, multi‑modal world model for autonomous‑driving simulation. It jointly generates RGB, semantic, depth, and 3‑D occupancy maps from a vehicle trajectory, uses normalized Plücker ray‑maps for precise control, and provides occupancy‑based dense rewards for closed‑loop policy evaluation. The repo includes installation steps (including a manual patch to `transformers`), data preparation instructions for nuScenes, pretrained checkpoint download links, and scripts for inference, out‑of‑distribution testing, and staged training.

  • Project
    TIGER-AI-Lab/VLM2Vec

    A unified framework for training and evaluating omni-modality embedding models that produce fixed-dimensional vectors for text, images, videos, audio, and visual documents.

  • Project
    NVIDIA/cosmos-framework

    An end-to-end framework for training and serving omnimodal world models, including the Cosmos 3 family, designed to unify language, vision, audio, and action for Physical AI.

  • Project
    ashuoAI/SHUO-Canvas

    SHUO Canvas is a desktop, node‑based canvas that lets creators combine text, images, video and audio with AI generation (via OpenAI‑compatible, RunningHub, ComfyUI, etc.) to build scripts, storyboards, character‑swap videos, 360° panoramas, voice‑overs and 3‑D scenes—all within one visual workflow.