Image GPT

OpenAI has introduced Image GPT (iGPT), a model that applies the GPT-2 transformer architecture to sequences of pixels. The research demonstrates that a large transformer trained on next-pixel prediction can generate coherent image samples and learn visual features that are competitive with top unsupervised convolutional neural networks, proving that domain-specific architectural priors can be traded for computational scale.

Generative Pre-training from Pixels

iGPT treats images as 1-D sequences of pixels, applying the same domain-agnostic transformer architecture used for natural language in GPT-2. By training the model to predict the next pixel in a sequence, iGPT learns to understand 2-D image characteristics, such as object appearance and category, without the use of human-provided labels.

To manage the computational cost of dense attention over long sequences, OpenAI utilized the following approach:

  • Low Resolution: Training was conducted at resolutions of 32x32, 48x48, and 64x64.
  • Custom Color Palette: A 9-bit color palette was created to represent pixels, reducing the input sequence length by three times compared to standard RGB palettes while maintaining color fidelity.
  • Model Scales: Four versions were trained: iGPT-S (76M parameters), iGPT-M (455M), iGPT-L (1.4B), and iGPT-XL (6.8B).

Correlation Between Generation and Classification

The research establishes a direct link between a model's generative performance and its ability to perform image classification. OpenAI found that as the model's scale increased and training iterations continued, generative quality improved, which translated directly into higher feature quality for downstream tasks.

Feature Extraction and Depth

Feature quality is not uniform across the network. OpenAI observed that the best features for image classification lie in the middle of the network rather than the final layer. This suggests a two-phase operation:

  1. Contextualization: The model gathers information from surrounding pixels to build a contextualized image feature.
  2. Prediction: The model uses that feature to solve the conditional next-pixel prediction task.

Experimental Results and Benchmarks

iGPT was evaluated using linear probes (logistic regression on learned features) and full fine-tuning across several datasets.

Unsupervised Performance

On several datasets, iGPT-L (32x32) outperformed other supervised and unsupervised transfer algorithms using linear probes:

  • CIFAR-10: 96.3% accuracy (compared to 95.3% for SimCLR).
  • CIFAR-100: 82.8% accuracy (compared to 80.2% for SimCLR).
  • STL-10: 95.5% accuracy (compared to 94.2% for AMDIM).

On ImageNet, iGPT-XL (64x64) achieved 72.0% top-1 accuracy using 15,360 features from five layers, outperforming AMDIM, MoCo, and CPC v2, though it remained below SimCLR's 76.5%.

Comparison with BERT-style Pre-training

OpenAI tested a masked-pixel approach similar to BERT (masking 15% of pixels). While BERT-style models performed significantly worse on linear probes, they were competitive with generative models during full fine-tuning.

Semi-Supervised Learning

Using a simple linear probe on non-augmented images, iGPT-L outperformed Mean Teacher and MixMatch on low-data CIFAR-10, although it underperformed FixMatch.

Limitations and Computational Costs

Despite its performance, iGPT has significant practical limitations due to its lack of domain-specific priors:

  • Compute Intensity: iGPT-L required approximately 2,500 V100-days of training, whereas a similarly performing MoCo model requires roughly 70 V100-days.
  • Resolution Constraints: Because it uses a transformer rather than a convolutional encoder, iGPT is limited to low-resolution inputs. Scaling to higher resolutions would likely require a new architecture, such as a multiscale transformer.
  • Algorithmic Bias: As a generative model, iGPT can inherit and amplify biases present in its training data, potentially leading to unfair or non-representative image completions.

Conclusion

iGPT serves as a proof-of-concept that large-scale sequence transformers can learn excellent unsupervised representations in novel domains without hand-coded architectural knowledge. By trading 2-D spatial priors for computational scale, iGPT demonstrates that the simplicity and generality of the GPT architecture can be effectively extended from language to vision.

Sources