HarborYuan/ovsam

[ECCV 2024] The official code of paper "Open-Vocabulary SAM".

What it solves

Open-Vocabulary SAM (OVSAM) addresses the limitation of the Segment Anything Model (SAM) by enabling it to not only segment objects but also recognize them across thousands of classes in an open-vocabulary setting. It allows for simultaneous interactive segmentation and recognition.

How it works

OVSAM uses two knowledge transfer modules to combine the strengths of SAM and CLIP:

  • SAM2CLIP: Adapts SAM's segmentation knowledge into CLIP via distillation and learnable transformer adapters.
  • CLIP2SAM: Transfers CLIP's recognition capabilities into SAM to enhance its ability to identify objects.

Who it’s for

Computer vision researchers and developers working on interactive image segmentation and open-vocabulary object recognition.

Highlights

  • Open-Vocabulary Recognition: Capable of recognizing and segmenting twenty-thousand classes interactively.
  • Knowledge Transfer: Uses a dual-module approach to bridge the gap between segmentation and recognition models.
  • Conda-based Installation: Provides a detailed setup guide for PyTorch and OpenMMLab packages.
  • RWKV-SAM Integration: Includes a related project exploring high-efficiency alternatives to the standard SAM architecture.

Related

  • Project
    microsoft/aurora

    Aurora is a foundation model for the Earth system that predicts atmospheric variables like temperature, air pollution, and ocean waves by adapting a general-purpose pre-trained model to specialized forecasting tasks.

  • Project
    nghorbani/amass

    A unified database and toolkit for human motion capture, consolidating various marker-based datasets into a common framework for animation and deep learning training.

  • Project
    koide3/small_gicp

    A highly optimized, parallelized C++ library for fine point cloud registration (ICP, GICP, VGICP) designed for high-speed 3D alignment.

  • Project
    Xiaoqi-Zhao-DLUT/MSNet-M2SNet

    A medical image segmentation framework featuring MSNet and M2SNet architectures that use multi-scale subtraction to accurately isolate pathologies like polyps and lung infections.