Fine-tuning CLIP for Remote Sensing and Satellite Imagery

Researchers have fine-tuned OpenAI's CLIP model using satellite imagery and captions, demonstrating that domain-specific fine-tuning is necessary for high-performance retrieval in remote sensing. The resulting model significantly outperforms the baseline CLIP model in matching aerial photographs to their corresponding textual descriptions.

Technical Approach and Model Architecture

The project utilized a fine-tuned version of the openai/clip-vit-base-patch32 model. The architecture employs a dual-encoder system—a text encoder and an image encoder—that uses contrastive learning to map both modalities into a joint embedding space. In this space, matching image-caption pairs are pushed closer together, while dissimilar pairs are pushed further apart.

Training Infrastructure

Training was conducted using the following stack:

  • Frameworks: Flax and JAX (a linear algebra library with automatic differentiation).
  • Hardware: Google Cloud TPUs.
  • Optimization: The best-performing model used the Adam optimizer with a learning rate of 5e-6 and a batch size of 1024 (distributed as 128 across 8 TPU cores).

Dataset and Data Augmentation

To adapt the model to the unique visual characteristics of satellite imagery, the team used three primary datasets:

  • RSICD: The primary dataset, containing approximately 10,000 RGB images (224x224) from Google Earth, Baidu Map, MapABC, and Tianditu, with up to five captions per image.
  • UCM: Based on the UC Merced Land Use dataset, containing 2,100 images across 21 classes with five captions each.
  • Sydney: Containing 613 images of Sydney, Australia, across seven classes with five captions each.

Regularization Strategies

To prevent overfitting on these relatively small datasets, the team implemented two types of augmentation:

  • Image Augmentation: Utilized PyTorch Torchvision for random cropping, random resizing, color jitter, and random horizontal and vertical flipping.
  • Text Augmentation: Employed backtranslation using the Marian MT family of models. Captions were translated from English into French, Spanish, Italian, and Portuguese, then translated back to English to increase the variety of captions for images with fewer than five unique descriptions.

Loss plots indicated that image augmentation significantly reduced overfitting, and the combination of both image and text augmentation provided the furthest reduction in overfitting.

Evaluation Results

Evaluation was performed using a subset of the RSICD test set across 30 image categories. The model was tested by ranking 30 captions of the form "An aerial photograph of {category}" for each image and measuring the top-k accuracy.

Model-name k=1 k=3 k=5 k=10
baseline 0.572 0.745 0.837 0.939
Best Model (Adam, lr 5e-6) 0.883 0.968 0.982 0.998

The fine-tuned model achieved a significant performance boost over the baseline, particularly at k=1, where accuracy increased from 0.572 to 0.883.

Practical Applications and Capabilities

The fine-tuned CLIP model enables several remote sensing capabilities:

  • Text-to-Image Search: Searching large satellite image collections using textual queries (e.g., "beach," "airport," or specific man-made features).
  • Image-to-Image Search: Finding visually similar satellite images within a corpus.
  • Feature Localization: By dividing an image into patches and encoding them, the model can match a text vector to specific image patches to determine the probability of a feature's presence in a specific area.

Ethical Considerations

The authors note that while this technology can be used for social good—such as monitoring climate change or national defense—it also raises ethical concerns regarding its potential misuse for military and police surveillance by authoritarian states.

Future Research Directions

The team identified several areas for further development:

  • Image Captioning: Developing a sequence-to-sequence model combining a CLIP encoder with a GPT-3 decoder.
  • Dataset Expansion: Fine-tuning on additional image-caption pairs to further improve performance.
  • Generalization: Investigating how fine-tuning affects performance on non-RSICD data and the model's ability to classify categories outside its training set.

Sources