CLIP: Connecting Text and Images

OpenAI has introduced CLIP (Contrastive Language–Image Pre-training), a neural network that learns visual concepts from natural language supervision. By training on a vast variety of images and text pairs found on the internet, CLIP can perform a wide range of visual classification tasks without requiring task-specific training data, effectively enabling "zero-shot" capabilities similar to those found in GPT-2 and GPT-3.

Solving the Limitations of Standard Computer Vision

CLIP addresses three primary failures of traditional deep learning approaches to computer vision:

Reduction of Costly Datasets: Traditional models rely on manually labeled datasets (such as ImageNet, which required 25,000 workers to annotate 14 million images) that are expensive and limited in scope. CLIP instead utilizes publicly available text-image pairs from the internet.
Increased Flexibility: Standard models are typically restricted to the specific categories they were trained on. To adapt a standard model to a new task, practitioners must build a new dataset and fine-tune the model. CLIP can be adapted to nearly any visual classification task by simply providing the names of the visual concepts to the text-encoder.
Closing the Robustness Gap: There is often a significant gap between a model's performance on a benchmark and its performance in real-world "wild" deployments. OpenAI conjectures that this happens because models "cheat" by optimizing specifically for the benchmark. Because CLIP is evaluated on benchmarks without being trained on their specific data, its performance is more representative of real-world utility. In tests, CLIP closed this robustness gap by up to 75% while matching the original ResNet-50 performance on ImageNet zero-shot.

Technical Approach and Training Efficiency

CLIP is trained using a proxy task: given an image, the model must predict which text snippet out of a randomly sampled set of 32,768 was actually paired with that image. This forces the model to learn a broad array of visual concepts and associate them with their natural language names.

To optimize training compute, OpenAI implemented two key algorithmic choices:

Contrastive Objective: CLIP uses a contrastive objective to connect text with images. In small to medium scale experiments, this was found to be 4x to 10x more efficient at zero-shot ImageNet classification than an image-to-text approach.
Vision Transformer (ViT): The adoption of the Vision Transformer provided a further 3x gain in compute efficiency over a standard ResNet.

As a result, the best performing CLIP model was trained on 256 GPUs for two weeks.

Capabilities and Generalization

CLIP demonstrates high flexibility across more than 30 different datasets, including tasks such as OCR, action recognition in videos, geo-localization, and fine-grained object classification. In representation learning evaluations using linear probes, the best CLIP model outperformed the Noisy Student EfficientNet-L2 on 20 out of 26 different transfer datasets.

Limitations and Challenges

Despite its general capabilities, CLIP has specific weaknesses:

Abstract and Systematic Tasks: CLIP struggles with counting objects in an image or predicting spatial relationships (e.g., how close a car is in a photo), where performance is only slightly better than random guessing.
Fine-Grained Classification: The model is less effective than task-specific models when distinguishing between very similar categories, such as specific aircraft variants, flower species, or car models.
Out-of-Distribution Data: CLIP shows poor generalization to images not covered in its pre-training. For example, it achieved 88% accuracy on handwritten digits from the MNIST dataset, significantly lower than the human accuracy of 99.75%.
Prompt Sensitivity: Zero-shot classifiers can be sensitive to specific wording, sometimes requiring "prompt engineering" to achieve optimal performance.

Broader Impacts and Biases

Because CLIP allows users to design their own classifiers without task-specific data, the choice of labels can introduce or amplify biases. OpenAI found that when given a set of labels including race labels and egregious terms like "criminal" or "animal," the model classified people aged 0–20 into the egregious category at a rate of ~32.3%. Adding the class "child" to the labels reduced this rate to ~8.7%.

Regarding privacy and surveillance, CLIP achieved a top-1 accuracy of 59.2% for "in the wild" celebrity image classification when choosing from 100 candidates, and 43.3% when choosing from 1,000 candidates. OpenAI notes that while this is possible with task-agnostic pre-training, it is not competitive with production-level celebrity identification models.