bytedance/Sa2VA

Official Repo For Pixel-LLM Codebase: Sa2VA (PAMI-26), SAMTok (CVPR-26), VRT (Arxiv-25), SaSaSa2VA (1-st solution for LSVOS)

What it solves

It addresses the limitation of multimodal LLMs in performing dense, pixel-level understanding. While many models can describe an image, they often struggle to precisely locate and segment specific objects or regions within images and videos.

How it works

The project is centered around Sa2VA, a unified model that combines the Segment Anything Model 2 (SAM-2) with LLaVA (Large Language-and-Vision Assistant). This integration allows the model to perform referring segmentation, grounded conversation, and visual prompting across both images and videos. It supports various backbones including InternVL2.5/3 and Qwen2.5-VL/Qwen3-VL.

Who it’s for

Researchers and developers working on multimodal AI, specifically those focused on image and video segmentation, visual grounding, and the intersection of LLMs and computer vision.

Highlights

  • Unified Model: Combines SAM-2 and LLaVA for dense grounded understanding.
  • Multi-Task Support: Capable of referring segmentation, grounded conversation, and image/video chat.
  • Broad Backbone Compatibility: Works with InternVL and Qwen-VL series models.
  • Extended Ecosystem: Includes related projects like VRT for object-level reasoning and SAMTok for representing masks as tokens.