FineVideo Dataset Release

Hugging Face has introduced FineVideo, an open-source dataset designed to address the scarcity of high-quality, annotated video data. FineVideo consists of 43,000 videos spanning 3,400 hours, featuring rich metadata including descriptive narratives, scene splits, and QA pairs, making it a critical resource for training multimodal LLMs, video diffusion models, and computer vision classifiers.

Dataset Composition and Sourcing

FineVideo was derived from a massive initial pool of 1.9 million English-language videos from YouTube-Commons, a collection of CC-By licensed content. The curation process involved several stages of filtering to ensure high data quality:

  • Initial Filtering: The dataset was narrowed to English videos and transcripts, gathering metadata such as resolution, view counts, and closed captions.
  • Downloading: 1.8 million videos were successfully downloaded using a combination of cloud batch jobs (via ytdlp and S3) and video2dataset.
  • Dynamic Content Filtering: To ensure the dataset contained active content rather than static images or screen casts, Hugging Face applied two filters:
    • Word Density: Videos with fewer than 0.5 words per second in closed captions were removed to ensure audio dynamism.
    • Visual Dynamism: Using FFMPEG's freezedetect filter with high noise parameters, videos where more than 40% of segments were identified as static were discarded.

After these filters, the pool was reduced to 600,000 dynamic videos.

Video Categorization and Taxonomy

To maintain diversity, Hugging Face developed a custom multi-level taxonomy containing 126 fine categories. This taxonomy was bootstrapped using GPT-4o and refined by information scientists.

Videos were categorized using Llama 3.1 70B served via Text Generation Inference (TGI). The team discovered that removing existing YouTube tags and categories from the prompt significantly improved categorization quality by reducing bias toward YouTube's own metadata.

High-Density Annotation Pipeline

FineVideo provides timecode-level metadata, including activities, objects, and narrative arcs. The annotation process utilized a two-step pipeline to overcome the limitations of structured output in large models:

1. Free-Text Generation with Gemini 1.5 Pro

Gemini 1.5 Pro was used to generate detailed descriptions of characters, scenes, mood, narrative progression, and Q&A pairs. To maintain quality, the team implemented the following constraints:

  • Length Limit: Videos longer than 10 minutes were dropped, as sampling revealed a significant drop in annotation quality for longer content.
  • Content Selection: A custom algorithm balanced content categories, user engagement (likes, views, comments), and channel representation to select a final 4,000 hours of content within budget constraints.

2. Structured Alignment with GPT-4o

Because strict schema constraints often degrade LLM performance, the team generated free-text descriptions first and then used the Instructor library with GPT-4o to parse that text into a structured Pydantic schema. This ensured the richness of the Gemini descriptions while maintaining the machine-readability of the final dataset.

Fine Alignment and Quality Control

To ensure temporal accuracy, the team performed "Fine Alignment," matching the scene boundaries provided by Gemini 1.5 (which processes video at 1 frame per second) to the actual frames in the video (typically 25-29 fps). This process also served as an anomaly filter; videos where Gemini stopped providing useful data were discarded. Due to the length restrictions applied earlier, the failure rate was negligible (under 0.5%).

Future Applications

Hugging Face is currently using FineVideo to train a multimodal LLM and intends to release the model weights and training recipe to the community upon completion.

Sources