wildminder/ComfyUI-VibeVoice

ComfyUI‑VibeVoice – What it does

ComfyUI‑VibeVoice is a set of custom nodes that bring Microsoft’s VibeVoice text‑to‑speech (TTS) and automatic‑speech‑recognition (ASR) models into the visual‑programming environment ComfyUI. Once installed, the nodes appear under WMNodes/sound/tts and WMNodes/sound/asr and can be dropped into a ComfyUI graph just like any other image‑generation node.

Core capabilities

Feature	What it means
Long‑form, multi‑speaker TTS	Up to four different voices can be spoken in a single audio file.
Voice cloning	Any local .wav/.mp3 can be used as a reference so the model mimics that speaker’s timbre.
Hybrid scripts	Mix cloned voices with speakers that reuse an already‑provided reference in the same script.
ASR	The companion VibeVoice ASR node returns timestamped speaker segments from an input audio file.
Quantized weights	Supports GGUF, fp8, int8 and 4‑bit quantisation (or the original safetensors) to reduce VRAM usage.
Attention back‑ends	Choose between eager, sdpa, flash_attention_2 or sage for speed/compatibility.
VRAM management	Models are downloaded automatically into ComfyUI/models/tts/VibeVoice/ and loaded through ComfyUI’s built‑in VRAM manager.

Available models (downloaded on first run)

Model	Size	Use case
VibeVoice‑1.5B	3 GB	Standard long‑form TTS (high quality).
VibeVoice‑7B	17.4 GB	Higher‑capacity TTS for richer voices.
VibeVoice‑Realtime‑0.5B	1.5 GB	Real‑time, single‑speaker generation (uses cached .pt voice prompts).
VibeVoice‑ASR‑HF	17.4 GB	Speech‑to‑text with speaker timestamps.

Typical workflow

1. Add a node – VibeVoice TTS (or VibeVoice ASR).
2. Load a reference audio (optional) with a Load Audio node and connect it to speaker_1_voice … speaker_4_voice to clone that speaker. Leaving an input empty makes the speaker reuse an already‑provided reference.
3. Write the script in the text field using either:
   • [1] Hello… or
   • Speaker 1: Hello…
4. Queue the graph – the node returns a single AUDIO output containing the whole conversation.

Realtime mode (single‑speaker only)

• Select the VibeVoice‑Realtime‑0.5B checkpoint.
• Provide a voice‑preset .pt file (e.g., en‑Carter_man.pt) in ComfyUI/models/tts/VibeVoice/voices/.
• Speaker‑reference inputs are ignored; the model uses the cached prompt.
• Adjust cfg_scale (1.5‑1.8) and max_new_tokens if the output is cut short.

Installation

• Via ComfyUI Manager – search for ComfyUI‑VibeVoice and install.
• Manual – clone into ComfyUI/custom_nodes, run pip install -r requirements.txt, then restart ComfyUI.
• Optional extras:
  • sageattention for the sage backend (Windows wheels provided).
  • bitsandbytes for 4‑bit quantisation.
  • ComfyUI-VibeVoice[audio-extra] for extra audio utilities.

Extending / loading external weights

• A Load VibeVoice Model node can point to a local .gguf, .safetensors or .bin file, overriding the built‑in model_name.
• Side‑car JSON files (<weights>.config.json, <weights>.preprocessor.json, tokenizer.json) let you supply custom architecture or tokenizer configs.
• The loader works on CPU first, then hands the tensors to ComfyUI’s VRAM manager; quantised formats stream directly and avoid extra host‑RAM copies.

Limitations & known issues

• Realtime models are single‑speaker only – speaker tags are stripped.
• At least one reference audio is required for any cloned voice; the system cannot generate a completely zero‑shot voice.
• The experimental DynamicVRAM patch for dense BF16 checkpoints does not yet reduce host‑RAM usage (see the “Known defect” section in the README).

---

In short: if you already use ComfyUI for image generation and want to add high‑quality, multi‑speaker TTS or speech‑to‑text without leaving the node‑graph environment, ComfyUI‑VibeVoice provides a ready‑made bridge to Microsoft’s VibeVoice models, handling downloads, VRAM allocation, and a flexible scripting format.