Tencent-Hunyuan/AuK

AuK – Open‑Source Foundational Model for Speech Generation & Editing

What it is – AuK is a 1.5 B‑parameter foundation model released by Tencent‑Hunyuan for a wide range of speech‑related tasks. It can synthesize new speech, edit existing recordings, enhance audio quality, and separate sources, all driven by natural‑language instructions.

Key capabilities

• Zero‑shot TTS – generate speech in the voice of a reference audio clip.
• Instruct TTS – synthesize from a textual voice description (no reference needed).
• Content editing – replace, insert or delete spoken words, including lyric editing for singing.
• Acoustic editing – pitch, speed and volume adjustments.
• Paralinguistic editing – change emotion, timbre, remove accents, add/remove non‑verbal sounds, convert between normal speech and whisper.
• Enhancement & separation – denoise, dereverberate, isolate a speaker, extract music vocals, or separate multiple speakers.

All tasks share a single message‑based API: you supply an instruction (free‑form text) and optionally a source/reference audio file, and the model returns the edited or generated waveform.

Model variants

Variant	Size	Speed	Typical use
AuK (base)	1.5 B parameters	Full diffusion (configurable steps)	Highest quality generation/editing
AuK‑Flash	Same architecture, distilled	Fixed 4‑step inference, CFG = 0	Real‑time or low‑latency scenarios

Both are hosted on Hugging Face and ModelScope; weights can be downloaded with huggingface_hub or modelscope CLI.

Quick start

1. Clone & set up

   git clone https://github.com/Tencent-Hunyuan/AuK
   cd AuK
   # Choose uv or conda (Python 3.10)
   uv venv --python 3.10 && source .venv/bin/activate
   uv pip install -e "[gradio,train]"   # install core + optional extras
   

2. Download checkpoints

   pip install -U "huggingface_hub[cli]"
   hf download tencent/AuK --local-dir ./ckpts/AuK
   hf download tencent/AuK-Flash --local-dir ./ckpts/AuK-Flash
   hf download Qwen/Qwen2.5-Omni-3B --local-dir ./ckpts/Qwen2.5-Omni-3B
   

3. Run a CLI example (replace a phrase in an audio file)

   auk-infer \
     --audio assets/demo-input-audio/content-edit/content.wav \
     --instruction "Replace 'but accepting what we cannot have' with 'and living well with dreams unmet'." \
     --output out.wav \
     --gen_seconds 7.0
   

4. Launch the Gradio web UI (optional)

   auk-gradio \
     --base_ckpt ckpts/AuK/auk_base.safetensors \
     --flash_ckpt ckpts/AuK-Flash/auk_flash.safetensors \
     --qwen_path ckpts/Qwen2.5-Omni-3B \
     --base_device cuda:0 --flash_device cuda:1 \
     --dtype bf16
   

   The UI lets you pick the model, upload audio, type an instruction, and hear the result.

Python API – The library provides AukInfer which accepts a list of message dictionaries (mirroring the chat‑style format used for LLMs) and returns a NumPy audio array and sample rate. Example snippets for content editing and instruction‑only TTS are shown in the README.

Fine‑tuning – A lightweight training script (src/auk/train/train.py) accepts JSONL files where each entry contains the instruction, optional source audio, and the target audio. Dynamic batching is driven by the target duration, and the repo includes a shell wrapper (scripts/train.sh).

Ecosystem integrations

• ComfyUI node package (comfyui/ComfyUI-AuK) for visual workflow pipelines.
• Prompt Enhancer – a helper that uses an OpenAI‑compatible LLM (e.g., Tencent Cloud TokenHub) to turn free‑form requests into the precise auk-infer command.

Licensing & citation

• The code is released under the Apache‑2.0 license (see LICENSE).
• Model weights are provided under a separate model‑specific license on Hugging Face/ModelScope.
• Citation format is provided in the README (BibTeX entry for the technical report).

Who might use it

• Researchers exploring unified speech generation/editing pipelines.
• Developers building voice assistants, dubbing tools, or audio post‑production software.
• Content creators needing quick voice‑over generation or audio cleanup without recording new material.

---

Bottom line – AuK is a genuine, open‑source, large‑scale speech foundation model that unifies synthesis, editing, enhancement, and separation behind a single natural‑language interface, with both high‑quality and low‑latency variants, ready‑to‑run CLI, web UI, and Python API.