mbzuai-oryx/groundingLMM

[CVPR 2024 🔥] Grounding Large Multimodal Model (GLaMM), the first-of-its-kind model capable of generating natural language responses that are seamlessly integrated with object segmentation masks.

What it solves

GLaMM addresses the limitation of Large Multimodal Models (LMMs) in providing precise, pixel-level visual grounding. It enables a model to not only describe an image but also generate segmentation masks that precisely identify the objects mentioned in its natural language responses.

How it works

GLaMM is an end-to-end trained multimodal model that can process both image-level and region-of-interest inputs. It integrates natural language generation with object segmentation masks, allowing it to perform a unified task called Grounded Conversation Generation (GCG). This process allows the model to combine phrase grounding, referring expression segmentation, and vision-language conversations into a single interaction.

Who it’s for

Researchers and developers working on computer vision and natural language processing who need a model capable of detailed region-level understanding, pixel-level grounding, and conversational AI.

Highlights

  • Grounded Conversation Generation: A new task that creates image-level captions tied to segmentation masks.
  • GranD Dataset: A large-scale dataset featuring 7.5 million unique concepts grounded in 810 million regions with segmentation masks.
  • Versatile Applications: Capable of referring expression segmentation, region-level captioning, and high-quality image captioning.
  • Multi-granularity Interaction: Supports interaction at multiple levels, from whole images to specific regions of interest.

Related

  • Project
    Merserk/dlss5-visual-enhancer

    A Windows application for NVIDIA RTX GPUs that uses DLSS 5 and RTX Video technologies to upscale, enhance, and interpolate frames for images, videos, and live streams.

  • Project
    worm128/AI-YinMei

    A one-stop AI live-streaming platform that integrates LLMs, streaming TTS, and virtual avatars to create interactive AI bots for multiple streaming platforms.

  • Project
    jaredrhod/barehands

    A webcam-based hand-tracking interface that lets you manipulate notes, images, and 3D models spatially, designed to be wired into an AI agent as a visual and interactive body.

  • Project
    Tencent-Hunyuan/UniRL

    UniRL is an open‑source Python framework that applies a unified reinforcement‑learning loop to many multimodal generative models (LLMs, vision‑language models, image/video diffusion, and hybrid AR‑diffusion models). It provides four entry points, Hydra‑based configs, pluggable rollout engines, and distributed training via Ray and FSDP. The repo ships standard RL algorithms plus three novel team‑proposed methods (Flow‑DPPO, DRPO, CPPO) with tutorials. Supported models include Stable Diffusion 3, FLUX.2‑Klein, Qwen‑3, HunyuanVideo, and more. Documentation, example recipes, and a WeChat community are provided. Licensed under Apache‑2.0.