open-mmlab/FoleyCrafter

What it solves

FoleyCrafter addresses the challenge of creating realistic sound effects for silent videos. It generates audio that is both semantically relevant to the visual content and precisely synchronized with the actions occurring in the video.

How it works

FoleyCrafter is a video-to-audio generation framework. It leverages a text-to-audio base model (Auffusion) and enhances it with specific adapters to handle visual information. It uses a semantic adapter to align audio with visual content and a temporal adapter to ensure the sound effects are synchronized with visual cues.

Who it’s for

This tool is designed for filmmakers, game developers, and content creators who need to add immersive, lifelike sound effects to silent video footage without manual foley recording.

Highlights

• Synchronized Audio: Produces sound effects that align temporally with visual cues in the video.
• Prompt-Based Control: Supports both positive prompts to guide the generation (e.g., "noisy, people talking") and negative prompts to exclude specific sounds (e.g., "wind noise").
• Semantic Relevance: Ensures the generated audio is contextually appropriate for the visual scene.
• Gradio Interface: Includes a user-friendly web interface for easier interaction and testing.