lxtGH/OMG-Seg

What it solves

OMG-Seg and OMG-LLaVA aim to replace multiple specialized vision models with a single, unified model capable of handling a wide range of visual perception and reasoning tasks. Instead of using a Large Language Model (LLM) to orchestrate separate specialist models, these frameworks provide an end-to-end solution for pixel-level understanding and reasoning.

How it works

• OMG-Seg: A transformer-based encoder-decoder architecture that uses task-specific queries and outputs to handle over ten different segmentation tasks (including semantic, instance, and panoptic segmentation for both images and videos, as well as open-vocabulary and interactive segmentation).
• OMG-LLaVA: Combines a universal segmentation method as a visual encoder with an LLM. It integrates image information, perception priors, and visual prompts into tokens that the LLM uses to provide text responses and pixel-level segmentation results based on user instructions.

Who it’s for

Researchers and academic labs focusing on computer vision and multi-modal LLMs, particularly those interested in reducing computational overhead and simplifying the model architecture for dense prediction tasks.

Highlights

• Unified Architecture: Supports joint co-training for multiple dense prediction tasks in one shot.
• Universal Segmentation: OMG-Seg handles image, video, and open-vocabulary segmentation within a single model.
• Efficiency: OMG-Seg uses only 70M trainable parameters and can be reproduced on a single 32GB V100 or 40GB A100 GPU.
• End-to-End: Bridges image-level, object-level, and pixel-level reasoning and reasoning in a single codebase.