apple-aiml-research/ml-fastvit

What it solves

FastViT is designed to create high-performance vision transformers that are fast enough for real-time use on mobile devices, reducing the latency typically associated with Vision Transformers (ViTs) while maintaining high accuracy.

How it works

It uses a hybrid architecture that combines the strengths of Vision Transformers and traditional convolutional neural networks. The core mechanism is structural reparameterization, which allows the model to have a complex structure during training (unfused) for better learning, but be simplified into a streamlined, faster version (fused/reparameterized) for inference.

Who it’s for

This project is for AI researchers and developers building computer vision applications for mobile or edge devices where low latency and high efficiency are critical.

Highlights

• Mobile-Optimized: Benchmarked specifically on iPhone 12 Pro to ensure real-world mobile performance.
• Structural Reparameterization: Supports a transition from a training-time model to a fused inference-time model for speed.
• Model Zoo: Provides a variety of pre-trained models (T8 through MA36) with different accuracy-latency trade-offs.
• Knowledge Distillation: Offers models trained with knowledge distillation to further boost Top-1 accuracy on ImageNet-1K.
• Cross-Platform Export: Includes tools to export PyTorch checkpoints to CoreML for deployment on Apple devices.