MultiMatte 2026 background removal model improves SAM 3 segmentation with promptable matting

MultiMatte adds promptable alpha‑matting to SAM 3 and outperforms it on every benchmark split

MultiMatte, released by Feyn, modifies only 2.27 % of SAM 3’s 860 M parameters with LoRA adapters yet achieves a 0.901 S‑measure on the DIS‑VD benchmark, compared with SAM 3’s 0.667. The improvement shows that a small low‑rank update can turn a binary segmenter into a high‑quality matte generator while preserving the original text‑prompt alignment.

Promptable matting solves fuzzy‑boundary failures of binary masks

SAM 3 returns binary masks for objects described by a text prompt, which cannot represent translucent or fine structures such as hair. MultiMatte replaces the binary output with an alpha matte that assigns a continuous opacity value to each pixel, enabling accurate extraction of fuzzy boundaries. On five high‑resolution DIS splits, SAM 3’s S‑measure ranges from 0.649 to 0.703, whereas MultiMatte achieves 0.893 to 0.923.

Low‑rank fine‑tuning (LoRA) preserves SAM 3’s text alignment

MultiMatte is trained with Parameter‑Efficient Fine‑Tuning (PEFT) using LoRA (rank‑16 adapters) on the attention and MLP projections of every tower, including the CLIP text tower. Each targeted linear layer keeps its pretrained weight frozen while learning two small matrices that add a low‑rank update. The adapters are merged into the released weights, so inference requires no extra libraries.

Training data and objectives

  • Images: 19,953 diverse images (salient objects, camouflage, hair, marine scenes).
  • Steps: 14,000 training steps.
  • Losses: Focal loss + Dice loss, the same semantic segmentation objective used by SAM 3.
  • Prompt supervision: 4,949 images (24.8 % of the set) contain human‑written labels naming the target object, teaching the new matte head to use SAM 3’s existing text alignment.

Benchmark results show consistent gains

Split SAM 3 MultiMatte Δ S‑measure
DIS‑VD 0.667 0.901 +0.233
DIS‑TE1 0.667 0.901 +0.234
DIS‑TE2 0.703 0.923 +0.220
DIS‑TE3 0.685 0.921 +0.235
DIS‑TE4 0.649 0.893 +0.244
DAVIS‑S (no training sibling) 0.913 0.979 +0.066
DUT‑OMRON (no training sibling) 0.792 0.901 +0.109

All splits improve; the largest absolute gains appear on datasets without a sibling in the training mix, indicating strong generalization. Changes below 0.002 S‑measure are treated as measurement noise.

Prompt steering still adds value after fine‑tuning

Even after LoRA adaptation, providing a concept name improves performance. On DIS‑VD, a real concept name adds 0.150 S‑measure to SAM 3 without any gradient steps and still contributes 0.036 to MultiMatte, confirming that the prompt pathway survived the retraining.

Easy inference with the NoBg library

MultiMatte is distributed via the nobg Python package. The LoRA adapters are merged, so the model can be used directly:

from nobg import AutoModel, AutoProcessor

model = AutoModel.from_pretrained("feyninc/multimatte")
processor = AutoProcessor.from_pretrained("feyninc/multimatte")

# Default prompt (no concept name)
cutout = model.predict(processor, "photo.jpg")
cutout.save("output.png")

# Prompted removal (keep "the dog")
cutout = model.predict(processor, "photo.jpg", "the dog")

predict returns an RGBA cutout; setting return_type="tensor" yields the raw matte.

Community feedback highlights real‑world robustness

"I tried it with an image that most background remover tools fail on – a horse and fence with similar colours – and MultiMatte did it perfectly!" – zurtri

"Is the demo client‑side? It matches my current workflow with Preview.app → Tools → Remove Background." – peterldowns

"Perfect timing as remove.bg is being folded into Canva; this looks great!" – FlamingMoe

These comments confirm that MultiMatte handles challenging colour‑similar foreground/background cases and that users appreciate its ease of integration.

References

  1. Meta. SAM 3: Segment Anything with Concepts. arXiv:2511.16719, 2025.
  2. Hu et al. LoRA: Low‑Rank Adaptation of Large Language Models. arXiv:2106.09685, 2021.
  3. Thinking Machines Lab. LoRA Without Regret. 2025. https://thinkingmachines.ai/blog/lora/
  4. Lin et al. Focal Loss for Dense Object Detection. arXiv:1708.02002, 2017.
  5. Sargsyan & Navasardyan. FlowDIS: Language‑Guided Dichotomous Image Segmentation with Flow Matching. CVPR 2026. arXiv:2605.05077.

Sources

Related

  • Project
    facebookresearch/sam3

    SAM 3 (Segment Anything with Concepts) is Meta’s 848 M‑parameter foundation model that lets you prompt an image or video with free‑form text (or visual exemplars) and receive masks, boxes, and scores for *all* matching objects. It combines a DETR‑style detector and a SAM 2‑style tracker, introduces a presence token for fine‑grained prompt discrimination, and is trained on >4 M auto‑annotated concepts. The repo provides installation steps, example notebooks, and a new SA‑CO benchmark (270 K concepts) for evaluation.

  • Dispatch
    Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers

    Hugging Face published a guide on training and finetuning multimodal embedding and reranker models with Sentence Transformers, demonstrating how task-specific finetuning improves performance on retrieval tasks such as Visual Document Retrieval.

  • Dispatch
    FeyNoBg: State-of-the-Art Background Removal Model and NoBg Training Library

    FeyNoBg is a state-of-the-art background removal model that achieves top S-measure on four of eight benchmarks and within 2% of the leader on the rest, accompanied by the open‑source NoBg library for training and inference.

  • Dispatch
    Sentence Transformers 6.0 MultiVectorEncoder: Training and Finetuning Guide

    Hugging Face announced the MultiVectorEncoder model type in Sentence Transformers v6.0 and provided a complete recipe for finetuning a ColBERT‑style retriever that outperforms general‑purpose models on a medical retrieval benchmark.

  • Dispatch
    Sentence Transformers v6.0 Multi-Vector Encoder release

    Sentence Transformers v6.0 adds a MultiVectorEncoder model type for ColBERT‑style late‑interaction retrieval, enabling token‑level embeddings across text, images, audio, and video with higher retrieval quality at the cost of larger indexes.