JavisVerse/JavisDiT

What it solves

JavisDiT++ is designed to solve the challenge of Joint Audio-Video Generation (JAVG), specifically the difficulty of creating videos that are both semantically and temporally aligned with their accompanying audio based on text prompts.

How it works

The project uses a Diffusion Transformer (DiT) architecture to generate sounding videos. It employs several key technical innovations:

• Joint Self-Attention: Enables dense interaction between audio and video modalities during generation.
• Modality-Specific MoE (MS-MoE): Refines representations within each modality to improve quality.
• Temporally Aligned Rotary Position Encoding (TA-RoPE): Ensures fine-grained synchronization between audio and video tokens.
• AV-DPO: A Direct Preference Optimization technique used to align the generated output with human preferences for better quality and synchronization.

Who it’s for

This tool is for researchers and developers working on multimodal AI, specifically those focused on text-to-audio-video generation and the synchronization of different sensory modalities.

Highlights

• Unified Modeling: Combines audio and video generation into a single optimized framework.
• Multi-stage Training: Supports a pipeline from audio pretraining to audio-video SFT and final preference alignment via DPO.
• Preference Dataset: Includes the AV-DPO preference dataset for improving human-alignment.
• Evaluation Suite: Provides JavisBench for comprehensive evaluation of audio-video alignment and quality.