Multimodal Neurons in CLIP

OpenAI has identified "multimodal neurons" within the CLIP model that activate in response to a single concept regardless of whether it is presented as a photorealistic image, a sketch, or written text. This discovery suggests that CLIP utilizes a mechanism of abstraction similar to the human brain, explaining its ability to classify diverse visual renditions of the same object.

Multimodal Neurons and Semantic Invariance

CLIP contains neurons that respond to semantic clusters of ideas rather than specific visual features. This mirrors biological findings, such as the "Halle Berry" neuron in the human brain, which fires for photographs, sketches, and the written name of the actress.

In CLIP, OpenAI identified a "Spider-Man" neuron (Neuron 244 in the penultimate layer of CLIP RN50x4) that activates for:

  • Photorealistic images of Spider-Man in costume.
  • Images of actual spiders.
  • Comic book illustrations of Spider-Man.
  • The written text "spider."

Using feature visualization and dataset examples, researchers found that the majority of neurons in CLIP RN50x4 are interpretable and function as "multi-faceted neurons" that respond to multiple distinct cases at a high level of abstraction. These include neurons for emotions, animals, famous people, art styles, and even digital alterations.

Composition and Classification

CLIP performs classification by composing these high-level multimodal neurons. A sparse linear probe reveals that final classifications are often the result of combining several conceptual neurons.

  • Visual Composition: The classification of a "piggy bank" is achieved by combining a "finance" neuron and a "porcelain" neuron. Similarly, the "Spider-Man" neuron contributes to the classification of a "barn spider."
  • Textual Composition: For language, concepts behave almost linearly. For example, the model interprets the word "surprised" as a combination of "shock" and "celebration/hug."

However, this reductive approach can lead to omissions. The model's understanding of "intimate" involves a "soft smile" and "hearts" but explicitly subtracts "illness," failing to account for intimacy in the context of sickness.

The "Absent Concept" Problem

Not all capabilities of CLIP are represented by single, interpretable neurons. While CLIP can perform precise geolocation (down to specific neighborhoods like "Twin Peaks" in San Francisco), researchers could not find a specific "San Francisco" neuron. This suggests that some information is encoded as a direction or a complex manifold rather than a discrete neuron.

Vulnerabilities: Typographic Attacks

The model's reliance on high-level abstractions creates a new attack vector called "typographic attacks." Because CLIP's multimodal neurons generalize across literal and iconic representations, they can be triggered by images of text.

For example, the "finance" neuron responds to both piggy banks and the text "$$$". By adding the string "$$$" to an image of a Standard Poodle, the model can be fooled into classifying the dog as a piggy bank. These attacks are practical and can be executed using simple pen and paper by writing text on a physical object.

Bias and Overgeneralization

CLIP inherits biases from its internet-scale training data, which manifest as problematic associations within its neurons:

  • Regional Bias: A "Middle East" neuron is associated with terrorism, and an "immigration" neuron responds to Latin America.
  • Representational Harm: One neuron was found to fire for both gorillas and dark-skinned people.
  • Resolution Discrepancy: Neurons for countries like the US and India are well-defined, whereas neurons for African countries tend to fire for entire regions rather than specific nations.

OpenAI suggests that interpretability tools are essential for practitioners to preempt these biases before deployment, as they may be difficult to anticipate or measure through standard testing.

Research Tools and Model Release

To facilitate further research into multimodal systems, OpenAI has released the weights for CLIP RN50x4 and RN101. Additionally, the OpenAI Microscope catalog has been updated to include feature visualizations, dataset examples, and text feature visualizations for every neuron in CLIP RN50x4.

Sources