Zero-shot image segmentation with CLIPSeg

TL;DR

CLIPSeg is a zero-shot image segmentation model that allows users to segment objects in an image using either text descriptions or example images (visual prompting) without needing to train the model on specific categories. While it provides rough, low-resolution masks, it is highly versatile for robot perception, image inpainting, and generating pre-labels for fine-tuning more precise segmentation models.

How CLIPSeg Works: Leveraging CLIP Embeddings

CLIPSeg achieves zero-shot segmentation by building upon CLIP (Contrastive Language–Image Pre-training), a model developed by OpenAI. CLIP creates abstract representations, or embeddings, of images and text in a shared high-dimensional space where similar concepts are positioned close together.

To enable segmentation, CLIPSeg employs a Transformer-based decoder trained on the PhraseCut dataset, which contains over 340,000 phrases and corresponding masks. The technical architecture involves:

  • Frozen CLIP Backbone: The underlying CLIP model remains frozen during training.
  • Decoder Integration: The decoder takes the CLIP representation of the target image and the CLIP representation of the prompt (text or image).
  • Layer Utilization: The decoder utilizes not only the final CLIP representation but also outputs from several intermediate CLIP layers to generate a binary segmentation mask.

Key Capabilities: Text and Visual Prompting

CLIPSeg is distinguished by its ability to handle two types of prompts to identify objects for segmentation:

Text Prompting

Users can provide a text string (e.g., "pancakes" or "blueberries") as a prompt. The model uses the CLIP text embedding of these words to locate and mask the corresponding objects in the image.

Visual Prompting

CLIPSeg allows the use of an example image as a prompt instead of text. This is particularly useful for objects that are difficult to describe verbally, such as specific logos on clothing. To optimize visual prompting, the research suggests:

  • Cropping: The prompt image should be cropped to contain only the object of interest.
  • Background Modification: Blurring or darkening the background of the prompt image can slightly improve results.

Technical Limitations and Practical Applications

While powerful in its flexibility, CLIPSeg has specific technical constraints:

  • Resolution: The model operates on images of 352 x 352 pixels, resulting in low-resolution, "fuzzy" output masks that are not pixel-perfect.
  • Use Case: Due to the resolution limits, it is best suited for rough localization or as a starting point for further refinement.

Workflow for High-Precision Segmentation

Because CLIPSeg provides rough labels, it can be used to accelerate the creation of high-quality datasets. A recommended workflow is to use CLIPSeg to generate initial pre-labels, then refine those masks using a labeling tool like Segments.ai, and finally fine-tune a state-of-the-art segmentation model (such as SegFormer) on the refined data.

Implementation with Hugging Face Transformers

CLIPSeg is integrated into the ‹‹transformers‹‹ library. Implementation requires the CLIPSegProcessor and CLIPSegForImageSegmentation classes.

For text prompting, the processor handles both the text and the image. For visual prompting, the processor generates separate embeddings for the target image and the prompt image, which are then passed to the model as conditional_pixel_values.

Sources