smthemex/ComfyUI_Sonic

ComfyUI Sonic – Audio‑driven portrait animation for ComfyUI

What it does
Sonic is a research method that animates a still portrait from an audio track, letting the face “talk” or express emotions that match the sound. This fork packages the Sonic pipeline as a custom node for ComfyUI, a visual, node‑based interface for building generative‑AI workflows.

Key components

• Audio‑to‑latent models (audio2bucket.pth, audio2token.pth) that turn the waveform into a sequence of tokens the video model can understand.
• UNet video diffusion model (unet.pth) that generates the frames conditioned on those audio tokens.
• Face detector (yoloface_v5m.pt) to locate the portrait region.
• OpenAI Whisper‑tiny for speech transcription (used by the original Sonic pipeline).
• RIFE optical‑flow model (flownet.pkl) for temporal smoothing.
• Optionally a Stable Video Diffusion (SVD) checkpoint (svd_xt*.safetensors) for higher‑quality video generation.

How you use it

1. Clone the repo into ComfyUI/custom_node.
2. Install Python dependencies with pip install -r requirements.txt.
3. Download the model files listed in the README and place them under ComfyUI/models/sonic/ (and the SVD checkpoint under ComfyUI/models/checkpoints).
4. In ComfyUI’s graph editor, add the Sonic node.
   • Feed an audio file (or a Whisper‑generated transcript).
   • Set duration (seconds) to control how long the output video should be.
   • Adjust image_size if you run out of GPU memory; smaller values reduce the resolution.
   • The node outputs a short video of the portrait moving in sync with the audio.

What’s new in this fork

• Replaces torchaudio.save with soundfile.write to avoid an ABI mismatch on newer PyTorch builds.
• Fixes for machines that must explicitly use cuda:0.
• Work‑arounds for BF16 support, 12 GB GPU OOM on first run, and Apple‑MPS (Mac) compatibility.
• UI tweaks: “frame number” renamed to duration, batch‑size handling for non‑25 fps inputs, and support for non‑square output images.

Typical workflow example
The README shows three example workflow screenshots. A user loads a portrait image, connects an audio file, sets the desired duration, and runs the graph to obtain a short animated clip.

Who might find this useful

• Creators who want quick, AI‑driven lip‑sync or expressive avatar videos without writing code.
• Researchers experimenting with audio‑driven video diffusion who prefer a visual UI over command‑line scripts.
• Hobbyists using ComfyUI for other diffusion tasks who want to add an audio‑driven animation node to their pipelines.

Citations
If you publish work that uses this node, cite the original Sonic paper (arXiv:2411.16331) and the related RealTalk paper (arXiv:2406.18284).