Hugging Face Video Dataset Scripts

Hugging Face has released a suite of open video dataset scripts to standardize the creation of high-quality training data for video generation models. This tooling aims to bring the same level of established infrastructure to video datasets that currently exists for image generation, combining small-scale custom scripts with video2dataset for large-scale applications.

Three-Stage Video Data Pipeline

The video dataset pipeline is structured into three distinct stages: acquisition, pre-processing/filtering, and processing. This architecture is inspired by data pipelines used in Stable Video Diffusion and LTX-Video.

Stage 1: Acquisition

Acquisition relies on yt-dlp for downloading video content. To handle long-form content, the pipeline includes a "Video to Scenes" script that splits long videos into shorter, manageable clips.

Stage 2: Pre-processing and Filtering

Filtering is applied at both the frame and video levels to ensure data quality and safety:

  • Frame-level filtering:
    • Watermark Detection: Uses LAION-5B-WatermarkDetection to identify watermarked content.
    • Aesthetic Scoring: Uses improved-aesthetic-predictor to assign a quality score to frames.
    • NSFW Detection: Uses Falconsai/nsfw_image_detection to filter out inappropriate content.
  • Video-level filtering:
    • Motion Scoring: Uses OpenCV to predict the amount of motion within the entire video clip.

Stage 3: Processing

Processing focuses on generating descriptive metadata for the clips. The pipeline utilizes microsoft/Florence-2-large to perform several tasks on extracted frames, including <CAPTION>, <DETAILED_CAPTION>, <DENSE_REGION_CAPTION>, and <OCR_WITH_REGION>.

While Florence-2 is the default for frame-level captioning, the pipeline is flexible. Other captioners can be integrated, and entire videos can be captioned using models such as Qwen2.5-VL instead of individual frames.

Filtering Insights and Benchmarks

Applying restrictive filters can significantly reduce dataset size while increasing quality. In the creation of the finetrainers/crush-smol dataset, Hugging Face used Qwen2VL captions and filtered for pwatermark < 0.1 and aesthetic > 5.5. This process reduced a pool of 1,493 total videos down to 47.

Watermark and Aesthetic Filtering Observations

  • Watermark Detection: The pwatermark score is effective at detecting text, but cannot distinguish between a watermark overlay and in-scene text (e.g., a toy car's license plate). Hugging Face suggests that using an average score across frames with a threshold of 0.2 - 0.3 is more effective than requiring every single frame to be below a strict threshold.
  • Aesthetic Scoring: High aesthetic thresholds (e.g., > 5.5) may introduce bias, potentially filtering out useful data. The authors note that aesthetic scores are more effective as a "bad content" filter with a minimum threshold of 4.25 - 4.5 rather than a tool to find "perfect" content.

Practical Application: Fine-tuning CogVideoX-5B

To demonstrate the utility of these scripts, Hugging Face created two specialized datasets: Cakeify and Crush. These datasets were used to fine-tune the CogVideoX-5B model using the finetrainers library.

One resulting model, finetrainers/crush-smol-v0, can generate videos of objects being flattened by a hydraulic press based on detailed text prompts, such as a red candle being crushed into a flat shape on a metal platform.

Sources