TaoLiveAIGC/TaoMate

TaoMate – Real‑time Audio‑Video Digital‑Human Generation

What it is – TaoMate is a research‑grade system that can synthesize a talking digital human (avatar) in real time, producing synchronized high‑resolution video and speech from a sequence of text prompts. The repository contains the inference runtime, multi‑GPU launch scripts, and an interactive browser demo that runs a resident model worker.

Key components

• Generator – built on the LTX‑2.3 22‑B transformer (BF16) with a custom anchor‑guided memory bridge that keeps long‑form context across up to 12 prompt segments (≈ 1 minute of output).
• Text encoder – Gemma‑3 12 B IT model, used to embed the prompt text.
• Dialogue & planning – optional Gemma‑4 26 B A4B GGUF model for on‑the‑fly dialogue and prompt planning in the interactive demo.
• Audio – OpenAI Whisper‑tiny for speech‑to‑text (ASR) in the demo; the generator itself produces the audio track.
• Distributed inference – single‑GPU or two‑GPU launchers split the model across devices; the full demo uses four GPUs (generation, text conditioning, decoding, dialogue).

Hardware requirements

• Linux x86_64, Python 3.10
• NVIDIA GPU(s) with CUDA 12.8; 72 GB VRAM needed for the reference 512×768, 1‑minute batch (single‑GPU peak ≈ 66.8 GiB, two‑GPU ≈ 33.6 GiB each)
• ffmpeg, tmux, curl

Installation (summary)

# system deps
sudo apt-get update && sudo apt-get install -y build-essential cmake curl ffmpeg git tmux python3-venv

# repo setup
cd TaoMate
python3.10 -m venv .venv && source .venv/bin/activate
pip install --upgrade pip
pip install torch==2.8.0 torchvision==0.23.0 torchaudio==2.8.0 \
    --index-url https://download.pytorch.org/whl/cu128
pip install -r requirements.txt

Verify CUDA with python -c "import torch; print(torch.__version__, torch.cuda.device_count())".

Model weights – Not stored in the repo. You must download three checkpoints from Hugging Face:

1. TaoMate (model.pt) – the fine‑tuned generator.
2. LTX‑2.3 (ltx-2.3-22b-dev.safetensors) – the base transformer.
3. Gemma‑3 12 B IT (full directory) – text encoder. Follow the README commands that use hf download and set the environment variables MODEL_CKPT, BASE_MODEL_CKPT, and GEMMA_PATH to the absolute paths.

Running inference

• Batch mode – use the provided shell scripts run_taomate_1gpu.sh or run_taomate_2gpu.sh. Supply a JSON list of cases (each case = up to 12 prompt/seed pairs). Example input format is shown in the README.
• Output – generated videos are written as <case_id>.mp4 in the directory you set via OUTPUT_DIR.

Interactive demo

1. Install the extra dialogue model (Gemma‑4 26 B GGUF) and build a CUDA‑enabled llama-server from ggml-org/llama.cpp.
2. Download Whisper‑tiny locally.
3. Launch the full stack with scripts/inference/start_interactive_avatar_stack.sh. The script starts:
   • a dialogue server (Gemma‑4),
   • a resident TaoMate worker (GPUs 0‑1),
   • a decoding/ASR worker (GPU 2),
   • a web service (GPU 3).
4. Open http://127.0.0.1:7860/ to interact with the avatar in a browser.
5. Stop everything with scripts/inference/stop_interactive_avatar_stack.sh.

Repository layout

apps/interactive_avatar/   # demo UI and worker code
configs/inference/         # example prompt JSON files
ltx_*/*                    # core transformer, cache, and decoder code
scripts/inference/         # launchers for batch and demo
taomate/inference/         # high‑level inference wrapper

License – Apache 2.0 for the code. Model weights retain their original licenses (LTX‑2.3, Gemma, Whisper, etc.).

Who might use this – Researchers exploring real‑time digital‑human synthesis, developers building avatar‑based chat or streaming services, and anyone needing a reference implementation of long‑form, synchronized audio‑video generation on high‑end GPUs.