Soul-AILab/SoulX-FlashTalk

SoulX-FlashTalk is the first 14B model to achieve sub-second start-up latency (0.87s) while maintaining a real-time throughput of 32 FPS on an 8xH800 node.

What it solves

SoulX-FlashTalk enables the real-time, infinite streaming of audio-driven avatars. It addresses the challenge of creating high-quality talking head animations from audio input that can run continuously without degradation in performance or quality.

How it works

The project utilizes a technique called "Self-Correcting Bidirectional Distillation" to achieve its streaming capabilities. It is built upon base models such as InfiniteTalk and Wan, and incorporates distillation techniques from DMD and Self-forcing++ to optimize the generation process for real-time performance.

Who it’s for

This tool is designed for developers and researchers working on digital humans, virtual avatars, and real-time interactive video systems.

Highlights

  • Infinite Streaming: Capable of generating continuous audio-driven avatar animations.
  • Real-Time Performance: Optimized for high-speed inference, with specific support for multi-GPU setups (e.g., 8xH800) for maximum speed.
  • Large Model Scale: Features a 14B parameter model for high-fidelity output.
  • Flexible Deployment: Supports single-GPU inference with CPU offloading to reduce VRAM requirements.

Related

  • Project
    lipku/LiveTalking

    A real-time interactive streaming engine for digital humans that synchronizes audio and video for lifelike AI-driven dialogues.

  • Project
    Alibaba-Quark/LiveAvatar

    Live Avatar is a research‑grade system that turns spoken audio (plus an optional reference image or text prompt) into a continuous, real‑time video of a talking avatar. Built on a 14 B diffusion backbone (WanS2V‑14B) with a LoRA fine‑tune, the authors achieve ~45 FPS streaming on multi‑GPU H800 hardware and can generate videos longer than 10 000 seconds via block‑wise autoregressive processing. The repo provides Conda‑based installation, multi‑GPU and single‑GPU inference scripts, optional FP8 quantisation and JIT compilation for speed, and a Gradio UI. Model weights are hosted on Hugging Face under Apache 2.0.

  • Project
    fudan-generative-vision/Hallo-Live

    Hallo-Live is a real-time text-driven framework that generates synchronized audio and video for digital avatars using a causal dual-stream Diffusion Transformer.

  • Project
    PunithVT/ai-avatar-system

    AvatarAI is an open‑source, production‑grade system that turns any portrait photo and a short voice sample into a real‑time, lip‑synced talking avatar. It streams Whisper‑transcribed speech to a selectable LLM (Claude, GPT‑4o, or local Ollama/vLLM), feeds the generated text to a multilingual TTS (Chatterbox/XTTS), and renders video frames with MuseTalk V1.5. The stack includes Next.js front‑end, FastAPI back‑end, PostgreSQL, Redis, Celery, Prometheus, and Docker‑Compose, with optional one‑click AWS GPU deployment. Features include zero‑shot voice cloning, barge‑in interruption, multi‑LLM support, JWT auth, conversation memory summarisation, and optional face‑restoration via GFPGAN.

  • Project
    anliyuan/Ultralight-Digital-Human

    A lightweight digital human model designed for real-time talking-head generation on mobile devices using personalized training from short videos.