modelscope/FunClip

What it solves

FunClip is an automated video clipping tool that allows users to extract specific segments of a video based on speech content or speaker identity. It eliminates the manual effort of scrubbing through long videos to find specific quotes or speakers to create shorter clips.

How it works

The tool uses a pipeline of AI models to process video files:

1. Speech Recognition (ASR): It leverages Alibaba's FunASR Paraformer series (including Paraformer-Large and SeACo-Paraformer for hotword customization) to transcribe the audio and predict precise timestamps for the text.
2. Speaker Identification: It integrates the CAM++ model to recognize different speakers, allowing users to clip segments based on who is talking.
3. AI-Assisted Clipping: It can integrate with LLMs (via gateways like OrcaRouter or Cheaper Inference) or video understanding models (like TwelveLabs Pegasus) to analyze transcripts or visuals and automatically suggest highlight segments.
4. User Interface: A Gradio-based web interface allows users to upload videos, select text segments from the transcription, and generate the final clips with subtitles.

Who it’s for

Content creators, video editors, and researchers who need to quickly isolate specific moments in video recordings based on what was said or who spoke, particularly for those working with Chinese and English audio.

Highlights

• Industrial-grade ASR: Uses Paraformer-Large for high-accuracy Chinese speech recognition and timestamp prediction.
• Speaker-based Clipping: Ability to trim videos based on auto-recognized speaker IDs.
• LLM Integration: Supports AI-driven highlight detection using external LLMs or content-aware video understanding via TwelveLabs Pegasus.
• Hotword Customization: Enhances recognition of specific names or entities using SeACo-Paraformer.
• Flexible Deployment: Can be run locally as a Gradio service or used via command-line interface.
• Automatic Subtitles: Generates SRT files for both the full video and the extracted clips.