tue-mps/eomt

[CVPR 2025 Highlight] Official code and models for Encoder-only Mask Transformer (EoMT).

What it solves

EoMT simplifies image segmentation by removing the need for complex, task-specific decoders or adapters. It demonstrates that a plain Vision Transformer (ViT) can perform high-quality segmentation without the architectural overhead typically required by state-of-the-art methods.

How it works

The Encoder-only Mask Transformer (EoMT) repurposes a pre-trained ViT to jointly encode image patches and segmentation queries as tokens. By treating both as tokens within the same transformer architecture, it eliminates the need for a separate decoder, allowing the model to output segmentation masks directly from the encoder.

Who it’s for

Computer vision researchers and developers working on image segmentation (panoptic, instance, and semantic) who want a faster, more minimalist architecture that maintains high accuracy.

Highlights

  • Minimalist Architecture: Uses only a plain ViT with no adapters or decoders.
  • High Performance: Achieves accuracy similar to SOTA methods while being up to 4× faster (e.g., with ViT-L).
  • Broad Support: Compatible with DINOv2 and DINOv3 backbones.
  • Versatile: Supports panoptic, instance, and semantic segmentation tasks across datasets like COCO, ADE20K, and Cityscapes.
  • Integration: Available via Hugging Face Transformers.

Related

  • Project
    HUANGLIZI/LViT

    LViT is a multimodal framework that combines language and vision transformers to improve the accuracy of medical image segmentation in CT scans and other medical imagery.

  • Project
    Intellindust-AI-Lab/EdgeCrafter

    EdgeCrafter provides compact Vision Transformers (ViTs) optimized for edge devices, enabling high-performance object detection, instance segmentation, and pose estimation with low latency.

  • Dispatch
    Fine-Tuning Vision Transformer (ViT) for Image Classification with Hugging Face Transformers

    Hugging Face provides a guide on fine-tuning the Vision Transformer (ViT) model for image classification using the datasets and transformers libraries, demonstrating the process with the beans dataset.

  • Project
    lucidrains/vit-pytorch

    A comprehensive PyTorch library implementing the original Vision Transformer (ViT) and dozens of its architectural variants for image classification.

  • Project
    lucidrains/x-transformers

    A concise and feature-rich Transformer library that implements a variety of experimental architectural optimizations and research-backed features for text and vision models.