facebookresearch/sam3

What is SAM 3?

SAM 3 (Segment Anything with Concepts) is Meta’s newest “foundation” model for prompt‑able segmentation of images and videos. It builds on the earlier Segment Anything models (SAM 1/2) but adds the ability to segment every instance of an open‑vocabulary concept that you describe with a short text phrase or a few example patches. In practice you can give the model a prompt like “all red cars” or “people wearing hats” and it will return masks, bounding boxes and confidence scores for every matching object in the scene, for both still images and video streams.

---

Why it matters

• Open‑vocabulary segmentation – SAM 3 can understand hundreds of thousands of noun phrases (the SA‑CO benchmark has 270 K unique concepts), far beyond the closed‑set categories of older detectors.
• Unified image + video pipeline – A single model contains a DETR‑style detector (text‑conditioned) and a SAM 2‑style transformer tracker, so you can use the same weights for single‑image inference, interactive refinement, or multi‑frame video tracking.
• Scalable data engine – The team automatically annotated >4 M unique concepts, creating the largest high‑quality open‑vocabulary segmentation dataset to date, which fuels the model’s strong performance (≈75‑80 % of human level on the new SA‑CO benchmark).
• Presence token & decoupled design – A special token helps the model discriminate between very similar prompts (e.g., “player in white” vs. “player in red”), and the detector‑tracker split reduces task interference, allowing the 848 M‑parameter model to scale efficiently.

---

Getting started (quick‑start code)

# Image example – text prompt
from sam3.model_builder import build_sam3_image_model
from sam3.model.sam3_image_processor import Sam3Processor

model = build_sam3_image_model()          # loads the default checkpoint (you need access on HF)
processor = Sam3Processor(model)
processor.set_image("my_photo.jpg")
out = processor.set_text_prompt(state=processor.state, prompt="all dogs")
# out contains masks, boxes, scores

# Video example – start a session and add a prompt
from sam3.model_builder import build_sam3_video_predictor

vid = build_sam3_video_predictor()
resp = vid.handle_request({"type": "start_session", "resource_path": "my_video.mp4"})
sess = resp["session_id"]
resp = vid.handle_request({
    "type": "add_prompt",
    "session_id": sess,
    "frame_index": 0,
    "text": "people playing soccer"
})
outputs = resp["outputs"]  # masks, boxes, scores for each frame

The repository ships with Jupyter notebooks that walk through these steps, batched inference, and even a SAM 3 Agent that can chain multiple prompts to solve more complex queries.

---

Core components

Component	Role
Detector	DETR‑style, conditioned on text, geometry, and exemplar patches. Finds objects that match the prompt.
Tracker	Re‑uses the SAM 2 transformer encoder‑decoder to propagate masks across video frames and support interactive refinement.
Presence token	Helps the model tell apart closely related textual prompts.
Vision encoder	Shared backbone for both detector and tracker, keeping the model size at 848 M parameters.

---

Benchmarks & performance

Task	Metric (higher = better)	SAM 3	Human
Image instance segmentation (SA‑Co/Gold)	cgF1	54.1	72.8
Image box detection (LVIS)	AP	48.5	–
Video segmentation (SA‑V test)	cgF1	30.3	53.1
Video tracking (SA‑V pHOTA)	pHOTA	58.0	70.5

These numbers show SAM 3 closing the gap to human performance while supporting a vastly larger vocabulary of concepts.

---

How to obtain the model

1. Request access to the checkpoints on the Meta Hugging Face repo (facebook/sam3).
2. Authenticate with hf auth login (or set HF_TOKEN).
3. Install the package (see the Installation section) and run the examples.

---

Who built it?

A large, cross‑disciplinary team from Meta Superintelligence Labs (core contributors, interns, and project leads) collaborated on the paper, code, data engine, and benchmark. The README lists over 40 contributors, spanning research, engineering, and product.

---

License & contribution

The code is released under the SAM License (a custom Meta license). Contributions are welcomed via the usual pull‑request workflow; see CONTRIBUTING.md and the code‑of‑conduct for details.

---

TL;DR

• SAM 3 = a single model that can detect, segment, and track anything you can describe with a short text phrase or a few visual examples.
• Works on images and videos, supports open‑vocabulary prompts (hundreds of thousands of concepts).
• Comes with a large automatically‑annotated dataset, a new presence‑token architecture, and state‑of‑the‑art results on the SA‑CO benchmark.
• Get the model from Hugging Face after requesting access, install via pip install -e ., and start experimenting with the provided notebooks.