Tencent-Hunyuan/AuK

AuK – Open‑source foundation model for speech generation & editing

What it is

• A 1.5 B‑parameter diffusion‑based model that can generate speech from text, edit existing audio, and perform enhancement or source separation – all driven by natural‑language instructions.
• Two checkpoints are released:
  • AuK (base) – high‑quality generation, configurable inference steps.
  • AuK‑Flash – a distilled 4‑step version for fast, low‑latency inference.

Key capabilities

Category	Tasks (natural‑language prompt)
Speech generation	Zero‑shot TTS (copy voice from reference), Instruct TTS (voice described only)
Content editing	Replace/insert/remove spoken words, lyric rewriting
Acoustic editing	Pitch shift, speed change, volume adjustment
Paralinguistic editing	Emotion, timbre, de‑accent, add/remove breaths/laughs, whisper ↔ normal speech
Enhancement & separation	Denoising, dereverberation, speech/music separation, target‑speaker extraction

All tasks share a single CLI/API where you pass an --instruction string and optional --audio reference.

How to get started

1. Clone & set up – git clone https://github.com/Tencent-Hunyuan/AuK && cd AuK then create a Python 3.10 env with either uv or conda (see README). Install the core package or add extras ([gradio], [comfyui], [train]).
2. Download weights – use huggingface_hub or modelscope CLI to pull tencent/AuK and optionally tencent/AuK-Flash plus the Qwen2.5‑Omni‑3B encoder.
3. Run a quick inference – e.g.:

   auk-infer \
     --audio assets/demo-input-audio/zero-shot-tts/ref.wav \
     --instruction "Say: 'Hello world' with the same voice" \
     --output out.wav \
     --gen_seconds 2.0
   

   The same command works for editing, enhancement, etc.; just change the instruction.
4. Optional helpers –
   • Prompt Enhancer (src/auk/infer/pe.py) uses an OpenAI‑compatible LLM to turn a free‑form request into a ready‑to‑run auk-infer command.
   • Gradio UI (auk-gradio) provides a web interface with model selector, CPU offload, and optional cloud ASR.
   • ComfyUI nodes let you embed AuK in visual workflows.
   • Distillation pipeline (PR #22) lets you train your own AuK‑Flash.

Performance & hardware

• Base AuK needs ~24 GiB VRAM for bf16 inference; enabling --cpu_offload cuts that to ~16 GiB (≈32 % saving).
• AuK‑Flash runs with fixed 4 diffusion steps (CFG = 0) for real‑time use.
• Supports CUDA, Apple‑silicon (MLX), and offline GGUF inference via audio.cpp.

Community & ecosystem

• Integrated with vLLM‑Omni and SGLang‑Omni for serving on a single H100.
• Used as the baseline for the ICASSP 2027 Audio Editing Challenge.
• Demo spaces on HuggingFace and ModelScope let you try the model without any code.

Where to learn more

• Project website: https://auk-project.github.io/
• Technical report (arXiv): https://arxiv.org/abs/2609.08936
• Full instruction cookbook: docs/COOKBOOK.md
• Model cards on HuggingFace and ModelScope for download links.

---

AuK is a genuine open‑source speech foundation model, offering a unified, instruction‑driven interface for a wide range of generation, editing, and enhancement tasks.