Fine-Tuning Pixtral-12B for Satellite Imagery Classification

Fine-tuning Pixtral-12B on satellite imagery significantly improves its ability to classify complex visual scenes, increasing overall accuracy from 0.56 to 0.91. This demonstrates that domain-specific adaptation via Low-Rank Adaptation (LoRA) is more effective than prompt engineering for specialized visual domains.

LoRA Fine-Tuning for Efficient Model Adaptation

Low-Rank Adaptation (LoRA) allows for the efficient adaptation of large vision-language models (VLMs) by injecting small, trainable rank-decomposition matrices into the model's weights. This method avoids the need to modify the full model, making it more resource-efficient than full retraining.

LoRA is particularly useful when prompt engineering or few-shot examples are insufficient. While complex prompts can be difficult to maintain and produce inconsistent results, LoRA-based fine-tuning uses a small set of curated examples to steer the model more reliably toward specialized knowledge, specific tones, or domain-specific vocabulary.

Application to Satellite Imagery

Satellite imagery is a specialized visual domain used in government, defense, agriculture, and climate science for tasks such as tracking deforestation and monitoring environmental change. Because these images contain unique patterns and semantics, general-purpose vision models often struggle to extract reliable insights without specific specialization.

Case Study: The Aerial Image Dataset (AID)

To evaluate the impact of fine-tuning, Mistral AI used the Aerial Image Dataset (AID), a public domain benchmark for classifying satellite images into detailed scene categories. The dataset includes 30 classes, including nuanced distinctions such as "dense residential" versus "medium residential."

Baseline Performance of Pixtral-12B

Using a traditional classification setup with 8,000 training samples and 2,000 test samples, the base Pixtral-12B model achieved reasonable baseline results using a system prompt that enforced a structured JSON output. However, the model faced two primary limitations:

  1. Ambiguous Class Differentiation: The model struggled with classes that had subtle visual differences. For example, it incorrectly classified both "Playground" and "Stadium" as "Stadium" because it failed to capture the presence of seats surrounding the sports field.
  2. Hallucinations: Because the language model is not explicitly constrained to the label set, it occasionally hallucinated non-existent or invalid class names, with a hallucination rate of 5%.

Fine-Tuning Process and Implementation

Pixtral-12B was fine-tuned using Mistral's fine-tuning API and LaPlateforme UI. The strategy involved providing the assistant response with the correct label for a given system prompt and input image.

Key hyperparameter recommendations include:

  • Learning Rate: Start small to avoid overshooting optimal weights.
  • Batch Size: Use a size that fits computational resources for stable gradients.
  • Epochs: Start with a single epoch and monitor for overfitting.

On LaPlateforme, the engine automatically computes the optimal batch size based on dataset size and internal benchmarks.

Results and Performance Gains

Fine-tuning led to a substantial increase in classification metrics across all classes. The key improvements include:

  • Accuracy: Overall accuracy increased from 0.56 to 0.91.
  • Hallucinations: Hallucinations were reduced from 5% to 0.1%.
  • Cost-Efficiency: These results were achieved with a limited budget of $10 or less and a relatively small dataset of 8,000 samples across 30 classes.

Broader Implications for Specialized VLMs

The success of Pixtral-12B on satellite imagery suggests that LoRA fine-tuning is a scalable and cost-effective approach for other highly specialized visual data that are underrepresented in general VLM training sets. Potential applications include medical image captioning, transcription of ancient manuscripts, and detailed reports from surveillance imagery.

Sources

Related

  • Dispatch
    Mistral AI Model Customization and Fine-Tuning Services

    Mistral AI has introduced model customization options via a new SDK, serverless fine-tuning on la Plateforme, and custom training services to enable domain-specific model optimization.

  • Dispatch
    Pixtral Large Release Notes

    Mistral AI has released Pixtral Large, a 124B open-weights multimodal model that achieves state-of-the-art performance on MathVista, DocVQA, and VQAv2 while maintaining the text capabilities of Mistral Large 2.

  • Dispatch
    Pixtral 12B Release Notes

    Mistral AI has released Pixtral 12B, a natively multimodal model that combines a new 400M parameter vision encoder with a 12B parameter decoder to deliver high-performance multimodal reasoning without compromising text capabilities.

  • Dispatch
    Using LoRA for Efficient Stable Diffusion Fine-Tuning

    Hugging Face has integrated Low-Rank Adaptation (LoRA) into the diffusers library, enabling faster Stable Diffusion fine-tuning with significantly lower VRAM requirements and model weights as small as 3 MB.

  • Dispatch
    Comparing RoBERTa, Llama 2, and Mistral for Disaster Tweet Classification with LoRA

    A comparative study reveals that the smaller RoBERTa model outperforms Llama 2 and Mistral 7B in binary classification of disaster tweets when fine-tuned using Low-Rank Adaptation (LoRA).