facebookresearch/MetaCLIP

NeurIPS 2025 Spotlight; ICLR2024 Spotlight; CVPR 2024; EMNLP 2024

What it solves

Meta CLIP addresses the limitations of English-centric Contrastive Language-Image Pre-training (CLIP) models. Specifically, it tackles the lack of large-scale non-English data curation pipelines and the "curse of multilinguality," where adding multiple languages often degrades performance in English compared to English-only models.

How it works

Meta CLIP provides a comprehensive recipe for scaling CLIP to worldwide data. This includes specialized data curation, modeling, and training strategies that allow English and non-English data to mutually benefit and elevate each other's performance, achieving state-of-the-art multilingual performance.

Who it’s for

It is designed for researchers and developers building multimodal AI that needs to understand images and text across a wide variety of global languages.

Highlights

  • Worldwide Scaling: Scales CLIP to global data, overcoming the performance trade-offs between English and multilingual capabilities.
  • Extensive Model Zoo: Offers a wide range of pre-trained models, including Meta CLIP 1 and Meta CLIP 2, with various ViT architectures (S, M, B, L, H, bigG) and resolutions.
  • Distilled Models: Provides distilled versions of Meta CLIP 2 for more efficient deployment.
  • OpenCLIP Integration: Closely adheres to the original OpenAI CLIP setup to facilitate rigorous scientific ablation studies.
  • Multilingual Support: Uses tokenizers like facebook/xlm-v-base to handle worldwide text input.

Related

  • Project
    mlfoundations/open_clip

    An open-source implementation of OpenAI's CLIP for training and large-scale contrastive language-image pre-training, featuring a wide array of pretrained models and support for audio and generative captioning.

  • Project
    OFA-Sys/Chinese-CLIP

    A Chinese-language version of the CLIP model trained on 200 million image-text pairs for cross-modal retrieval and zero-shot image classification.

  • Dispatch
    CLIP: Connecting Text and Images

    OpenAI introduced CLIP, a neural network that learns visual concepts from natural language supervision to enable zero-shot image classification across diverse datasets.

  • Dispatch
    Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese

    Qwen has released Chinese CLIP, a vision-language model designed to overcome the cultural and linguistic limitations of English-centric CLIP models in cross-modal retrieval and image classification.

  • Dispatch
    Fine-tuning CLIP for Remote Sensing and Satellite Imagery

    A team of researchers fine-tuned OpenAI's CLIP model using the RSICD dataset and other satellite imagery to significantly improve text-to-image retrieval for remote sensing applications.