OmniCustom-project/OmniCustom

Official Implementation of 'OmniCustom: Sync Audio-Video Customization Via Joint Audio-Video Generation Model'

What it solves

OmniCustom addresses the challenge of synchronous audio-video customization. It allows users to generate a video that maintains a specific person's visual identity and an audio track that mimics a specific person's voice timbre, while the actual speech content is controlled by a text prompt.

How it works

The framework uses a joint audio-video generation model finetuned from OVI. It integrates several specialized components to handle different modalities:

  • Visual Identity: Extracts 512-D face embeddings using InsightFace and crops reference images via LivePortrait.
  • Voice Timbre: Extracts 256-D timbre embeddings using Naturalspeech 3.
  • Audio Generation: Utilizes an Audio VAE from MMAudio.
  • Video Generation: The video branch is initialized from Wan2.2.

Users provide a reference image, a reference audio clip, and a text prompt to guide the final output.

Who it’s for

Content creators and developers looking to create highly customized, synchronized talking-head videos where both the appearance and the voice are based on real-world references.

Highlights

  • Joint Generation: Synchronously produces both video and audio rather than treating them as separate steps.
  • Identity Preservation: Maintains visual identity from a reference image and vocal timbre from a reference audio clip.
  • Text-Driven Content: Speech content is freely specified via textual prompts.
  • Multi-Model Integration: Combines strengths from OVI, Wan2.2, MMAudio, and InsightFace.

Related

  • Project
    Correr-Zhou/OmniShow

    OmniShow is a multimodal video generation model that unifies text, reference images, audio, and pose conditions to create high-fidelity videos of humans interacting with objects.

  • Project
    k2-fsa/OmniVoice

    OmniVoice is a diffusion‑based, zero‑shot TTS model covering 600+ languages. It offers voice cloning, attribute‑driven voice design, and auto‑voice generation, with fast inference (RTF ≈ 0.025). Install via pip, load the model with `OmniVoice.from_pretrained`, and call `model.generate` (or use the provided CLI tools). Optional FlashInfer kernels give 2‑3× speedup. The repo includes training scripts, a Gradio demo, and a Colab notebook.

  • Project
    cclank/lanshu-create-ai-presenter-video

    A generalized AI digital human video production workflow that transforms scripts and images into synchronized presenter videos with subtitles and effects.

  • Project
    TIGER-AI-Lab/VLM2Vec

    A unified framework for training and evaluating omni-modality embedding models that produce fixed-dimensional vectors for text, images, videos, audio, and visual documents.

  • Project
    inclusionAI/Ming

    Ming-flash-omni 2.0 is an open-source omni-MLLM that unifies multimodal perception and generation across text, image, audio, and video in a single MoE-based architecture.