OpenMOSS/MOVA

A foundation model that generates synchronized video and audio in a single model

What is MOVA?

MOVA (MO​SS V​ideo A​udio) is an open‑source foundation model that creates synchronized video and audio in a single forward pass. It can take a text prompt (and optionally a reference image) and output a short video where the spoken words match the lip movements and the background sounds fit the scene.

Core ideas

Feature Why it matters
Native bimodal generation No separate text‑to‑video + text‑to‑audio pipelines – the model produces both modalities together, avoiding error accumulation.
Asymmetric dual‑tower architecture Uses a pre‑trained video encoder and a pre‑trained audio encoder that talk to each other through bidirectional cross‑attention, giving rich cross‑modal interaction.
Lip‑sync & sound‑effects State‑of‑the‑art multilingual lip‑synchronization and environment‑aware audio, as shown on the Verse‑Bench benchmark.
Fully open‑source Model weights, inference code, training pipelines, LoRA fine‑tuning scripts, evaluation suite, and a public API are all released.
Hardware flexibility Supports GPU, NVIDIA H100/RTX 4090, Ascend NPU, and off‑loading strategies to run on limited VRAM.

Getting started (quick‑start)

  1. Create the environment
    conda create -n mova python=3.13 -y
conda activate mova
pip install -e .
  2. Download a checkpoint (e.g., 360p) from Hugging Face:
    hf download OpenMOSS-Team/MOVA-360p --local-dir /path/to/MOVA-360p
  3. Run inference – single‑person example:
    export CP_SIZE=1
export CKPT_PATH=/path/to/MOVA-360p
torchrun --nproc_per_node=$CP_SIZE \
    scripts/inference_single.py \
    --ckpt_path $CKPT_PATH \
    --cp_size $CP_SIZE \
    --height 352 --width 640 \
    --prompt "A man in a blue blazer …" \
    --ref_path ./assets/single_person.jpg \
    --output_path ./data/samples/single_person.mp4 \
    --seed 42 \
    --offload cpu
    Replace the prompt and --ref_path for multi‑person scenes.

How to fine‑tune (LoRA)

  • Choose a config under configs/training/ (low‑resource, accelerate, or accelerate + FSDP for 8‑GPU).
  • Install training extras: pip install -e ".[train]" (adds torchcodec, bitsandbytes).
  • Launch with the provided shell scripts, e.g.:
    bash scripts/training_scripts/example/low_resource_train.sh
  • All LoRA hyper‑parameters (rank, target modules, optimizer, off‑load strategy) are in the config files.

Evaluation

  • The repository ships an evaluation/ folder with 11 metrics covering audio quality, lip‑sync, AV alignment, etc.
  • Pre‑computed benchmark results on Verse‑Bench and a human‑rated Arena benchmark (732 samples) are provided for reproducibility.

Ecosystem integrations

Integration What it gives you
SGLang Day‑0 high‑throughput inference via sglang generate or sglang serve.
ComfyUI Community node comfyui-mova for visual‑programming pipelines (both I2VA and T2VA).
API Hosted service at studio.mosi.cn; apply for a key to generate videos without local hardware.
Streamlit workflow End‑to‑end UI that can auto‑generate a first‑frame, rewrite prompts, and call MOVA.

Resources

  • Model weights – Hugging Face collections: MOVA-360p, MOVA-720p.
  • Technical report – arXiv: 2602.08794 (Feb 2026).
  • Demo videos – see the README or the website https://mosi.cn/models/mova.
  • Community – Discord, X (Twitter), Feishu groups linked in the badge bar.
  • Citation – provided in the README for academic use.

TL;DR

MOVA is a publicly released, dual‑tower video‑audio diffusion model that can generate short, lip‑synced videos from text (and an optional image). The repo includes everything needed to run inference on a single GPU, fine‑tune with LoRA, evaluate results, and plug into popular tools like SGLang, ComfyUI, or a hosted API.

Related

  • Project
    dsta022/Loop-Engineering-for-VLA

    Loop Engineering for VLAnything is an end‑to‑end Python toolkit that lets you record, merge, audit, enrich, and iteratively improve multimodal robot‑learning datasets (RGB, RGB‑D, and future visuo‑tactile) for vision‑language‑action models. It provides a conservative, side‑car‑first quality audit, plug‑in semantic evaluators, language‑annotation pipelines, and a closed‑loop feedback loop for policy improvement, all compatible with the LeRobot data format and publishable to Hugging Face.

  • Project
    ModelTC/LightX2V

    LightX2V is an open‑source inference framework for fast image and video generation (T2V, I2V, T2I, I2I, T2AV, etc.). It supports many state‑of‑the‑art models (MiniMax‑H3, Wan, HunyuanVideo, Qwen‑Image, etc.) and provides speed‑up techniques such as 4‑step distilled LoRAs, FP8/NVFP4 quantization, tensor/sequence parallelism, and disaggregated deployment on a wide range of hardware. Benchmarks show up to 3.9× faster inference than competing frameworks on H100 GPUs. The project offers Docker images, pip install, Gradio/ComfyUI front‑ends, extensive docs, and an online demo.

  • Project
    Alibaba-Quark/LiveAvatar

    Live Avatar is a research‑grade system that turns spoken audio (plus an optional reference image or text prompt) into a continuous, real‑time video of a talking avatar. Built on a 14 B diffusion backbone (WanS2V‑14B) with a LoRA fine‑tune, the authors achieve ~45 FPS streaming on multi‑GPU H800 hardware and can generate videos longer than 10 000 seconds via block‑wise autoregressive processing. The repo provides Conda‑based installation, multi‑GPU and single‑GPU inference scripts, optional FP8 quantisation and JIT compilation for speed, and a Gradio UI. Model weights are hosted on Hugging Face under Apache 2.0.

  • Dispatch
    Gemma 4 release: open‑source multimodal models with on‑device support

    Gemma 4, a new open‑source multimodal family from Google DeepMind, is released on Hugging Face with image, audio, video support, up to 256 K context, and day‑0 compatibility with transformers, llama.cpp, MLX, Rust, and WebGPU.

  • Project
    tong-io/tongflow

    TongFlow is an open‑source visual studio that lets you build multi‑modal generative AI pipelines by connecting simple “add”, “transform”, and “combine” nodes. It supports text, image, video, audio, 3‑D and documents, offers a rich plugin ecosystem (official API plugins, routers, and GPU/CPU compute plugins), and can be used via a tiny desktop client or self‑hosted backend. Licensed under AGPL‑3.0.