GPT-4o Vision Fine-Tuning API Release

OpenAI has introduced vision fine-tuning for GPT-4o, enabling developers to customize the model using both images and text. This capability allows for significant improvements in specialized visual tasks where text-only fine-tuning is insufficient, such as enhanced visual search, autonomous vehicle object detection, and medical image analysis.

Technical Implementation of Vision Fine-Tuning

Vision fine-tuning for GPT-4o follows a process similar to text-based fine-tuning. Developers prepare image datasets in a specific format and upload them to the OpenAI platform.

Key technical details include:

  • Data Requirements: Performance improvements can be achieved with as few as 100 images, though larger volumes of text and image data can drive higher performance.
  • Model Support: The capability is supported on the gpt-4o-2024-08-06 model snapshot.
  • Input Format: Datasets use a JSON structure where messages can contain both text and image_url types, mirroring the format used in OpenAI's Chat endpoints.

Real-World Performance Gains

Collaborations with early partners demonstrate that vision fine-tuning provides substantial performance uplifts over the base GPT-4o model across various domains:

Mapping and Infrastructure (Grab)

Grab used 100 examples to teach GPT-4o to localize traffic signs and count lane dividers. This resulted in:

  • A 20% improvement in lane count accuracy.
  • A 13% improvement in speed limit sign localization.

Enterprise Automation (Automat)

Automat utilized screenshots and unstructured documents to improve RPA (Robotic Process Automation) agents:

  • UI Element Localization: By training the model to locate UI elements via natural language descriptions, the success rate increased from 16.60% to 61.67% (a 272% uplift).
  • Information Extraction: Training on 200 images of unstructured insurance documents yielded a 7% lift in F1 score.

Digital Content Creation (Coframe)

Coframe fine-tuned GPT-4o with images and code to generate branded website sections. This approach improved the model's ability to maintain consistent visual style and correct layout by 26% compared to the base model.

Safety, Privacy, and Data Control

Vision fine-tuning is governed by OpenAI's Enterprise privacy commitments. Key protections include:

  • Data Ownership: Fine-tuned models remain under the developer's control, and the developer retains full ownership of their business data.
  • Training Restrictions: OpenAI does not train its base models on the inputs or outputs used in the vision fine-tuning service without explicit permission.
  • Monitoring: OpenAI employs automated safety evaluations and monitors usage to ensure adherence to usage policies.

Availability and Pricing

Vision fine-tuning is available to all developers on paid usage tiers.

Pricing Structure (Post-October 31, 2024)

  • Training: $25 per 1M tokens.
  • Inference (Input): $3.75 per 1M tokens.
  • Inference (Output): $15 per 1M tokens.

Image inputs are tokenized based on image size and priced at the same per-token rate as text inputs. Note that OpenAI offered 1M free training tokens per day through October 31, 2024.

Sources