apple-aiml-research/ml-fastvlm

This repository contains the official implementation of "FastVLM: Efficient Vision Encoding for Vision Language Models" - CVPR 2025

What it solves

FastVLM addresses the high computational cost and latency associated with vision encoding in Vision Language Models (VLMs), particularly when processing high-resolution images. It aims to reduce the time-to-first-token (TTFT) and the overall size of the vision encoder without sacrificing accuracy.

How it works

The project introduces FastViTHD, a hybrid vision encoder that optimizes how images are processed to output fewer tokens. By reducing the number of tokens generated during the encoding phase, the model significantly speeds up the initial response time (TTFT) and requires less memory than traditional vision encoders.

Who it’s for

This is designed for researchers and developers building multimodal AI applications, especially those targeting resource-constrained environments like mobile devices (iOS) or Apple Silicon hardware.

Highlights

  • Significant Speedup: The smallest variant achieves 85x faster Time-to-First-Token (TTFT) compared to LLaVA-OneVision-0.5B.
  • Efficiency: Features a vision encoder that is 3.4x smaller than competing models of similar scale.
  • High Performance: Larger variants using Qwen2-7B outperform Cambrian-1-8B while maintaining a 7.9x faster TTFT.
  • Mobile Ready: Includes a demo iOS app to showcase real-time performance on mobile hardware.

Related

  • Project
    xushengfeng/eSearch

    A cross-platform screen search tool that integrates screenshotting, offline OCR, translation, and reverse image search to extract and act on screen information.

  • Project
    leejet/stable-diffusion.cpp

    A lightweight pure C/C++ implementation for running image and video diffusion models locally, supporting a wide range of hardware backends and model architectures.

  • Project
    EleutherAI/gpt-neox

    GPT‑NeoX is an open‑source, Megatron‑based library for training billion‑parameter language models on multi‑GPU clusters. It adds DeepSpeed‑style optimizations, supports many launchers (Slurm, MPI, etc.), works on NVIDIA and AMD GPUs, and includes modern features like flash attention, Mixture‑of‑Experts, and preference‑learning fine‑tuning. Ideal for research labs with large compute budgets; for inference‑only use, Hugging Face `transformers` is recommended.

  • Project
    flyteorg/flyte

    Flyte 2 is an open‑source Python framework for defining, orchestrating and serving ML pipelines and AI agents. It lets you write ordinary Python functions as tasks, compose them into workflows, and run them locally or on a future Kubernetes‑native backend. The project includes a CLI, a developer‑focused TUI, and first‑class support for FastAPI model serving.