apple-aiml-research/ml-mobileclip

📦 What is MobileCLIP?

MobileCLIP (and its newer version MobileCLIP2) are compact, fast image‑text models designed for zero‑shot classification and captioning on devices with limited compute, such as smartphones. They are trained with a multi‑modal reinforced training scheme that mixes large‑scale image‑text data (DataCompDR, DFNDR) with synthetic captions generated by CoCa models. The goal is to match the accuracy of heavyweight CLIP‑style models while using 2‑5× fewer parameters and much lower latency, especially on iPhone hardware.

---

🎯 Key Features & Highlights (from the README)

Feature	Detail
Model families	MobileCLIP‑S0…S4, MobileCLIP‑B, MobileCLIP‑L/14 and the newer MobileCLIP2‑S0…S4, MobileCLIP2‑B, MobileCLIP2‑L/14
Speed / size	Smallest variant (MobileCLIP‑S0) is 4.8× faster and 2.8× smaller than OpenAI’s ViT‑B/16 while keeping similar zero‑shot performance. MobileCLIP2‑S4 runs at ~20 ms (image + text) on an iPhone 12 Pro Max, delivering accuracy comparable to much larger models.
Accuracy	MobileCLIP‑B (LT) reaches 77.2 % top‑1 on ImageNet‑1k zero‑shot; MobileCLIP2‑S4 hits 81.9 % with only 13 B seen samples.
Training data	Trained on the DataCompDR‑1B dataset (for MobileCLIP) and on the DFNDR dataset (for MobileCLIP2).
Open‑source tooling	Inference, training, and evaluation scripts are provided. Models are also compatible with OpenCLIP via a small patch.
Mobile demo	An iOS app (ios_app/) showcases real‑time zero‑shot classification on‑device.
Model releases	Pre‑trained checkpoints are hosted on HuggingFace (e.g., apple/MobileCLIP2‑S0).

---

🚀 Getting Started (quick‑start)

1. Create a conda environment

   conda create -n clipenv python=3.10
   conda activate clipenv
   pip install -e .
   

2. Download a checkpoint (example for the smallest MobileCLIP2 model)

   hf download apple/MobileCLIP2-S0   # stores the .pt file locally
   

3. Run a one‑liner inference

   import torch, open_clip
   from PIL import Image
   from mobileclip.modules.common.mobileone import reparameterize_model
   
   model_name = "MobileCLIP2-S0"
   model_path = "/path/to/MobileCLIP2-S0.pt"
   model, _, preprocess = open_clip.create_model_and_transforms(
       model_name, pretrained=model_path)
   tokenizer = open_clip.get_tokenizer(model_name)
   model.eval()
   model = reparameterize_model(model)   # required for batch‑norm based models
   
   img = preprocess(Image.open("docs/fig_accuracy_latency.png").convert("RGB")).unsqueeze(0)
   txt = tokenizer(["a diagram", "a dog", "a cat"])
   with torch.no_grad(), torch.cuda.amp.autocast():
       img_feat = model.encode_image(img)
       txt_feat = model.encode_text(txt)
       img_feat = img_feat / img_feat.norm(dim=-1, keepdim=True)
       txt_feat = txt_feat / txt_feat.norm(dim=-1, keepdim=True)
       probs = (100.0 * img_feat @ txt_feat.T).softmax(dim=-1)
   print("Label probs:", probs)
   

   This prints the probability for each of the three text prompts.

---

📚 Training & Evaluation

• Training – The repo includes a training/ folder that contains scripts to train MobileCLIP/2 models using the OpenCLIP codebase. The README points to a separate repository (ml-mobileclip-dr) for generating the large‑scale multi‑modal reinforced datasets.
• Evaluation – Zero‑shot ImageNet‑1k evaluation can be run with:

  python eval/zeroshot_imagenet.py --model-arch mobileclip_s0 \
      --model-path /path/to/mobileclip_s0.pt
  

  For full 38‑dataset benchmarking, follow the instructions in the datacomp repo.

---

📱 iOS Demo

The ios_app/ directory contains a minimal iOS application that loads a MobileCLIP model and performs real‑time classification on camera frames. This showcases the model’s on‑device latency claims.

---

📦 Model Zoo (selected checkpoints)

Model	Params (M)	Latency (ms) img+txt	ImageNet‑1k Zero‑Shot
MobileCLIP2‑S0	11.4 + 63.4	1.5 + 3.3	71.5
MobileCLIP2‑S2	35.7 + 63.4	3.6 + 3.3	77.2
MobileCLIP2‑B	86.3 + 63.4	10.4 + 3.3	79.4
MobileCLIP2‑S4	321.6 + 123.6	19.6 + 6.6	81.9
MobileCLIP‑B (LT)	86.3 + 63.4	10.4 + 3.3	77.2

All checkpoints are downloadable from the HuggingFace collection linked in the README.

---

🧩 CoCa Caption Models

The repo also ships CoCa models that were used to generate synthetic captions for the DFNDR‑2B dataset. They can be loaded via OpenCLIP in the same way as the CLIP models, and a short example is provided for caption generation.

---

📜 License

• Code – MIT License
• Model weights – Apple ML Research Model Terms of Use (a permissive research‑only license)
• Data – CC‑BY‑NC‑ND 4.0 (non‑commercial, no‑derivatives)

---

📖 Citation

If you use MobileCLIP or MobileCLIP2 in research, cite the two papers listed in the README (CVPR 2024 and TMLR 2025).

---

TL;DR

MobileCLIP is a family of tiny, fast CLIP‑style vision‑language models optimized for mobile devices. The repository provides ready‑to‑use pretrained checkpoints, training scripts (via OpenCLIP), evaluation tools, and an iOS demo, all under permissive licenses.