PABannier/sam3.cpp

What it solves

Running state-of-the-art image and video segmentation models (like Meta's SAM series) typically requires a heavy Python environment, PyTorch, and CUDA GPUs. sam3.cpp provides a portable, high-performance C++ implementation that allows these models to run on CPUs and Apple Metal GPUs without the need for Python runtimes or heavyweight dependencies.

How it works

The project is a single-file C++ library built on top of the ggml tensor library. It implements inference for several model families, including SAM 2, SAM 2.1, SAM 3, and EdgeTAM. It supports 4-bit and 8-bit quantization to significantly reduce model size and memory usage. For SAM 3, it integrates a text encoder and DETR decoder to enable text-prompted detection, while other models focus on point and box-based segmentation. It also includes a memory bank for tracking objects across video frames.

Who it’s for

Developers and researchers who need to integrate high-quality image and video segmentation into C++ applications, particularly those targeting edge devices, macOS (via Metal), or environments where installing a full Python/PyTorch stack is impractical.

Highlights

• Broad Model Support: Compatible with SAM 2, SAM 2.1, SAM 3, and EdgeTAM.
• Text-Prompted Detection: SAM 3 allows users to segment objects simply by typing a description (e.g., "cat").
• Extreme Portability: Zero dependencies beyond ggml and stb; written in C++14 with no exceptions or inheritance.
• Hardware Acceleration: Full support for Apple Metal GPU acceleration for backbones and transformer decoders.
• Efficient Quantization: Reduces model sizes drastically (e.g., EdgeTAM down to 15 MB, SAM 2.1 Tiny to 22 MB).
• Video Tracking: Supports object tracking across video frames using a memory bank.