modelscope/FunClip

FunASR-powered video transcription, subtitle generation, and LLM-assisted clipping tool with a local Gradio UI.

What it solves

FunClip is an automated video clipping tool that allows users to extract specific segments of a video based on speech content, speaker identity, or AI-driven analysis. It eliminates the manual effort of scrubbing through long videos to find and cut specific quotes or scenes.

How it works

The tool uses a pipeline of speech and video understanding models:

  1. Speech Recognition (ASR): It leverages Alibaba's FunASR (specifically Paraformer, Fun-ASR-Nano, and SenseVoice) to transcribe video audio into text with timestamps.
  2. Speaker Identification: It uses the CAM++ model to recognize different speakers, allowing users to clip segments based on who is talking.
  3. Selection & Clipping: Users can manually select text segments from the transcription or use an LLM (like Qwen or GPT) to automatically identify relevant timestamps based on a prompt.
  4. Visual Understanding: Optionally, it integrates with TwelveLabs Pegasus to reason over both visuals and audio for more accurate highlight detection.

Who it’s for

Content creators, video editors, and researchers who need to quickly isolate specific spoken segments or speaker-specific clips from long-form video content.

Highlights

  • AI-Powered Clipping: Integrates LLMs to automatically identify and extract video segments based on natural language prompts.
  • Speaker-Based Trimming: Ability to clip segments from a specific speaker using auto-recognized speaker IDs.
  • Multilingual Support: Supports Mandarin, English, Japanese, and various Chinese dialects.
  • Hotword Customization: Allows users to specify entity words or names to improve ASR accuracy via SeACo-Paraformer.
  • Comprehensive Output: Automatically generates full video SRT subtitles and target segment SRT subtitles.