apple-aiml-research/ml-aim

What it solves

This project provides a family of large-scale vision encoders designed to overcome the limitations of traditional vision models by using autoregressive pre-training. It aims to create highly scalable and effective vision backbones that excel in multimodal understanding, object detection, and image recognition without requiring complex training pipelines.

How it works

AIM (Autoregressive Image Models) uses a multimodal autoregressive objective for pre-training. The latest version, AIMv2, scales effectively across different capacities (from 300M to 2.7B parameters) and resolutions (224px to 448px). It supports multiple backends including PyTorch, JAX, and MLX (for Apple silicon). The architecture allows for the use of frozen trunks for recognition tasks and supports native resolution processing via 2D sinusoidal positional embeddings.

Who it’s for

This project is for AI researchers and developers building multimodal applications, vision-language models, or high-performance image recognition systems that require a powerful, pre-trained vision backbone.

Highlights

• High Performance: Outperforms OAI CLIP and SigLIP on most multimodal benchmarks and DINOv2 on open-vocabulary object detection.
• Scalable Architecture: Offers models ranging from 0.3B to 2.7B parameters.
• Flexible Deployment: Native support for PyTorch, JAX, and MLX.
• Resolution Independence: Includes a specific checkpoint capable of processing a wide range of image resolutions and aspect ratios.
• Distilled Versions: Provides distilled ViT-Large models optimized for multimodal understanding.