bytedance/Sa2VA

Official Repo For Pixel-LLM Codebase: Sa2VA (T-PAMI-26), SAMTok (CVPR-26), VRT (Arxiv-25), SaSaSa2VA (1-st solution for LSVOS)

📚 What is Sa2VA?

Sa2VA (pronounced “Sa‑to‑VA”) is a research‑grade codebase from ByteDance that combines SAM‑2 – a state‑of‑the‑art segmentation model – with multimodal large language models (MLLMs) such as InternVL2.5/3 and Qwen‑VL. The goal is to give LLMs pixel‑level awareness of images and videos, enabling them to:

Refer to exact regions (referring segmentation) in a picture or video.
Carry on grounded conversations where the model can point to, mask, or edit visual content on the fly.
Support visual prompting – users can give a mask or a bounding box and the model will understand the intent.
Power image/video chat interfaces that feel as natural as text‑only chat.

The repository is organized as a collection of related projects, each extending the core Sa2VA model:

Project	Purpose	Notable Artefacts
Sa2VA	Core unified model (SAM‑2 + MLLM).	Supports InternVL2.5/3, Qwen2.5‑VL, Qwen3‑VL. Papers: IEEE TPAMI 2026, arXiv 2025.
VRT (Visual Reasoning Tracer)	Object‑level grounded reasoning built on Sa2VA.	Benchmark VRT‑Bench and training set VRT‑80k (HF dataset).
SAMTok	A “mask‑token” interface that lets any MLLM generate or understand masks using just two words.	CVPR 2026 paper, model zoo on HuggingFace.
SaSaSa2VA	Segmentation‑augmented extension that won 1st place in ICCV 2025 LSVOS Challenge (RVOS track).
Pixel‑SAIL	Single‑transformer approach to pixel‑level grounding.

🚀 How to get started

Install the lightweight dependency manager uv (one‑liner):
```
curl -LsSf https://astral.sh/uv/install.sh | sh
```
Create the environment (the script sets up a virtualenv in /tmp and links it back):
```
bash setup_env.sh                 # default: latest backbones
# bash setup_env.sh sa2va legacy  # for older InternVL2.5 etc.
```
Behind the scenes the pyproject.toml and uv.lock in projects/sa2va lock every transitive dependency, guaranteeing reproducible builds.
Configure secrets (e.g., HuggingFace token) by copying the template:
```
cp .env.example .env   # edit with your keys
```
Run a demo – after activation (source projects/sa2va/.venv/bin/activate), follow the README inside the specific project folder (e.g., projects/sa2va/README.md) to launch inference or training.

🎯 Typical use cases

Use case	What Sa2VA enables
Interactive visual chatbots	Users can ask “What is the object in the red box?” and the model returns a precise mask and description.
Video editing assistants	Provide a textual command like “blur the person in frame 12‑15” and the system isolates the exact pixels across frames.
Grounded visual reasoning	Combine object‑level masks with chain‑of‑thought prompting to answer “Why is the car moving faster than the bike?” using VRT‑Bench.
Cross‑modal content creation	Generate images with specific region constraints (e.g., “draw a cat where the blue circle is”).

📦 What’s inside the repo

projects/sa2va/ – core model code, training scripts, and inference utilities.
projects/vrt_sa2va/ – dataset loaders, benchmark evaluation scripts, and the VRT‑80k data preparation pipeline.
projects/samtok/ – token‑mask conversion logic and pretrained checkpoints.
setup_env.sh – one‑click environment bootstrap.
.env.example – placeholder for API keys.
Citation block – ready‑to‑copy BibTeX for the three main papers.

📚 Where to learn more

Paper (TPAMI 2026) – https://ieeexplore.ieee.org/document/11640960
arXiv pre‑print – https://arxiv.org/abs/2501.04001
Project page – https://lxtgh.github.io/project/sa2va
Model zoo – https://huggingface.co/collections/ByteDance/sa2va-model-zoo-677e3084d71b5f108d00e093
VRT benchmark page – https://harboryuan.github.io/visual-reasoning-tracer/
SAMTok page – https://zhouyiks.github.io/projects/SAMTok/

📝 How to cite

@article{sa2va,
  title={Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos},
  author={Yuan, Haobo and Li, Xiangtai and Zhang, Tao and Sun, Yueyi and Huang, Zilong and Xu, Shilin and Ji, Shunping and Tong, Yunhai and Qi, Lu and Feng, Jiashi and Yang, Ming‑Hsuan},
  journal={IEEE TPAMI},
  year={2026}
}

Bottom line: Sa2VA is a cutting‑edge research platform that gives large language models true pixel‑level vision, opening the door to tightly integrated visual‑language agents, grounded reasoning, and next‑generation multimodal assistants.