microsoft/XPretrain

XPretrain – Multi‑modality Pre‑training from Microsoft Research

What it is – XPretrain is a collection of research code and datasets for large‑scale pre‑training on video‑language and image‑language pairs. It bundles several models that were published at top conferences (CVPR 2022, NeurIPS 2022, ICLR 2023) and the accompanying HD‑VILA‑100M dataset, all coming from Microsoft’s Multimedia Search & Mining (MSM) group.

Key components

Modality	Model / Dataset	Paper (venue)	What it does
Video‑Language	HD‑VILA	CVPR 2022	High‑resolution video‑language pre‑training on the 100 M‑clip HD‑VILA‑100M dataset
Video‑Language	LF‑VILA	NeurIPS 2022	Pre‑training on long‑form videos (minutes‑long)
Video‑Language	CLIP‑ViP	ICLR 2023	Adapts the image‑language CLIP paradigm to video by adding a temporal pooling module
Image‑Language	Pixel‑BERT	arXiv 2020	End‑to‑end image‑language transformer trained from raw pixels
Image‑Language	SOHO	CVPR 2021 (oral)	Improves Pixel‑BERT with quantised visual tokens
Image‑Language	VisualParsing	NeurIPS 2021	Transformer‑based image‑language pre‑training with a parsing objective

What you can get

• Code for each model (training scripts, model definitions, evaluation utilities) located in sub‑folders (hd-vila/, LF-VILA/, CLIP-ViP/, etc.).
• Pre‑trained checkpoints (released together with the papers) that can be fine‑tuned on downstream tasks such as video captioning, video‑question answering, image‑text retrieval, etc.
• HD‑VILA‑100M dataset – a publicly released collection of high‑resolution video‑text pairs for large‑scale pre‑training.

Typical use cases

• Researchers building new video‑language models can start from the released checkpoints and fine‑tune on their own data.
• Practitioners needing a strong video‑text encoder for tasks like video search, caption generation, or multimodal retrieval.
• Academics studying the effect of resolution, temporal length, or token quantisation on multimodal learning.

Getting started

1. Clone the repo: git clone https://github.com/microsoft/XPretrain.git.
2. Install the required Python packages (the README points to a requirements.txt inside each sub‑folder).
3. Follow the README in the specific model directory (e.g., hd-vila/README.md) for data preparation, training, and inference commands.
4. For the dataset, download the HD‑VILA‑100M files from the hd-vila-100m/ folder (links provided there).

Community & contribution

• Contributions are accepted under Microsoft’s CLA; a bot will guide you through the process when you open a PR.
• Issues can be filed for help with the pre‑trained models; contact authors Bei Liu or Jianlong Fu for direct questions.

Why it matters XPretrain gathers several state‑of‑the‑art multimodal pre‑training pipelines in one place, making it easier for the community to reproduce results, benchmark new ideas, and leverage large‑scale video‑language representations without rebuilding the massive training infrastructure from scratch.