shengshu-ai/Vidu-S

Vidu S: Real-Time Interactive, Editable, and Spatial Video Generation

Vidu S – Real‑time Interactive Video Generation

What it is – Vidu S is a research‑grade system for generating and editing video on the fly. It comes in two versions:

  • Vidu S1 – a 540p, 42 FPS model that lets a user speak commands to a digital character (real‑person, anime, pet, etc.) and have the video change instantly.
  • Vidu S2 – an upgraded 720p, 25‑42 FPS engine that can drive high‑resolution avatars, edit live streams (style transfer, virtual try‑on, background swap), and output synchronized stereo streams for VR/AR displays.

Key technical ideas

  1. Self‑Replay Forcing – during training the model re‑feeds its own generated frames (after re‑noising) so it learns to keep consistency over long streams.
  2. TurboDiffusion / TurboServe – a suite of speed‑ups (efficient attention, low‑bit GEMM, kernel tweaks, multi‑GPU pipelining) that make real‑time inference possible on consumer‑grade GPUs.
  3. Live‑stream editing – reference images can be swapped in at any moment, letting the system change a person’s outfit, style, or even replace the whole character while preserving motion.
  4. Spatial video – Vidu S2 can emit left/right eye views for immersive head‑sets.

How you can use it

  • Try it online: a hosted demo is available at https://www.vidu.com/vidu‑stream.
  • API: a REST‑style API (documented in English and Chinese) lets developers send video frames, reference images, and textual or spoken instructions, and receive the generated stream back.
  • Agent integration: a vidu‑s‑api skill can be installed in Claude‑Code, Codex, OpenClaw, etc., enabling agents to call the Vidu S service directly.

Documentation & papers

  • Vidu S2 paper – arXiv:2609.11638 (covers the avatar, editing, and spatial video pipelines).
  • Vidu S1 paper – arXiv:2607.03118 (focuses on speech‑controlled generation).
  • Full user guides and quick‑start tutorials are hosted on Feishu (English & Chinese) and the platform’s API docs.

Who might benefit

  • Developers of live‑streaming or virtual‑host platforms that need on‑the‑fly avatar animation or video effects.
  • Researchers exploring diffusion‑based video generation, long‑horizon consistency, or low‑latency inference.
  • Creators wanting real‑time virtual try‑on or background replacement without pre‑rendering.

Citation – If you use Vidu S in a paper, cite the two arXiv pre‑prints provided in the README.

All details above are taken directly from the repository’s README; no extra features have been inferred.

Related

  • Project
    fudan-generative-vision/Hallo-Live

    Hallo-Live is a real-time text-driven framework that generates synchronized audio and video for digital avatars using a causal dual-stream Diffusion Transformer.

  • Project
    PKU-YuanGroup/Helios

    Helios is a 14B-parameter video generation model that produces minute-long, coherent videos in real time (19.5 FPS on a single H100) without relying on common anti-drifting or acceleration tricks. It offers three model variants (Base, Mid, Distilled), supports text/image/video-to-video generation, and includes memory-saving features like group offloading (~6GB VRAM) and multi-GPU context parallelism. It has broad ecosystem support (Diffusers, SGLang, vLLM-Omni, Ascend NPU) and was accepted by IEEE TPAMI.

  • Project
    Alibaba-Quark/LiveAvatar

    Live Avatar is a research‑grade system that turns spoken audio (plus an optional reference image or text prompt) into a continuous, real‑time video of a talking avatar. Built on a 14 B diffusion backbone (WanS2V‑14B) with a LoRA fine‑tune, the authors achieve ~45 FPS streaming on multi‑GPU H800 hardware and can generate videos longer than 10 000 seconds via block‑wise autoregressive processing. The repo provides Conda‑based installation, multi‑GPU and single‑GPU inference scripts, optional FP8 quantisation and JIT compilation for speed, and a Gradio UI. Model weights are hosted on Hugging Face under Apache 2.0.

  • Project
    Francis-Rings/StableAvatar

    StableAvatar is an open‑source diffusion‑based video transformer that generates arbitrarily long, high‑quality avatar videos driven by audio. It introduces a time‑step‑aware audio adapter, audio native guidance, and a sliding‑window fusion strategy to keep lip‑sync and identity stable over unlimited duration. The repo provides pretrained 1.3 B checkpoints, inference scripts (single‑GPU, multi‑GPU, Gradio, ComfyUI), and full training/LoRA code, with detailed data‑preparation instructions.

  • Project
    duixcom/Duix-Mobile

    An open-source SDK for deploying real-time interactive AI avatars on mobile and embedded devices with low latency and on-device execution.