OpenMOSS/MOSS-VL

What is MOSS‑VL?

MOSS‑VL is an open‑weight, 11‑billion‑parameter video‑language model family that can understand and talk about long videos. It comes in three variants:

Model	Main use case
MOSS‑VL‑Realtime	Real‑time interaction on a live video stream – the model watches, reasons, and can answer or stay silent at any moment.
MOSS‑VL‑Instruct	Offline, high‑quality video‑question‑answering and dialogue.
MOSS‑VL‑Base	A foundation checkpoint for further pre‑training or fine‑tuning.

All three share a cross‑attention architecture that separates visual encoding from language reasoning, uses absolute timestamps and a 3‑D rotary position embedding (XRoPE) so the model can ground its answers in both space and time.

---

Key Features

• Interruptible streaming – you can ask a question at any timestamp; the model replies immediately based on the frames it has seen so far.
• Proactive silence – when there is nothing worth saying, the model stays quiet instead of spitting out irrelevant text.
• Dynamic correction – as new frames arrive, the model can revise earlier statements.
• Long context – 256 K tokens of video‑language context, enough for hour‑long clips.
• Open‑source SOTA latency – the authors claim the fastest publicly available streaming video‑LLM.
• Quantized checkpoints – FP8 and NF4 versions (~24 GB) run on a single 24 GB GPU.
• Plug‑and‑play back‑ends – works with FlashAttention‑3, SGLang, and a specialized SGLang‑Omni server for multi‑stream serving.
• Fine‑tuning support – full‑parameter or LoRA training via the bundled scripts, plus out‑of‑the‑box integration with LLaMA‑Factory and ms‑swift.

---

Getting Started (Real‑time)

conda create -n moss_vl python=3.12 pip -y
conda activate moss_vl
pip install -r requirements.txt   # no‑build‑isolation as required by the repo

Run a video through the realtime pipeline (1 fps sampling, 256 frames max):

CUDA_VISIBLE_DEVICES=0 python inference/realtime/run_online_inference.py \
  --checkpoint OpenMOSS-Team/MOSS-VL-Realtime \
  --source video \
  --video path/to/example.mp4 \
  --sample-fps 1 \
  --playback-speed 1 \
  --max-frames 256

The script offers three integration levels:

1. Low‑level API – model.create_realtime_session(...) for custom frame‑by‑frame control.
2. Queue API – model.online_generate(...) for a worker‑based pipeline.
3. WebSocket service – --serve starts a FastAPI server that accepts JPEG/PNG frames or live video streams.

For production‑grade serving, the repo includes third_party/sglang-omni/, a SGLang‑based backend that schedules many concurrent streams and scales across GPUs.

---

Offline / Batch Usage

from transformers import AutoModelForCausalLM, AutoProcessor
import torch

ckpt = "OpenMOSS-Team/MOSS-VL-Realtime"
processor = AutoProcessor.from_pretrained(ckpt, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    ckpt, trust_remote_code=True, device_map="auto", torch_dtype=torch.bfloat16)

queries = [{
    "messages": [{"role": "user", "content": [
        {"type": "image", "image": "path/to/example.jpg"},
        {"type": "text", "text": "Describe this image."}
    ]}],
    "generate_kwargs": {"max_new_tokens": 256, "do_sample": False},
}]

with torch.no_grad():
    out = model.offline_batch_generate(processor, queries)
print([r["text"] for r in out["results"]])

The same checkpoint works for image‑only, video‑only, or mixed multimodal queries.

---

Ecosystem & Resources

• FlashAttention‑3 backend (third_party/flash-attention-src/) for fast cross‑attention.
• SGLang / SGLang‑Omni servers for low‑latency serving.
• Quantization guide (quantization/README.md) – FP8‑Dynamic and NF4 recipes, including KV‑cache quantization.
• Fine‑tuning – scripts under finetune/ plus compatibility with LLaMA‑Factory and ms‑swift.
• Benchmarks – streaming SOTA on PA@OmniMMI, TOMATO, StreamingBench; offline results shown in the repo’s figures.
• Demo – a full‑stack browser UI with ASR/TTS lives in third_party/realtime-demo/.
• Paper & Technical Report – arXiv:2608.15045 (MOSS‑VL Technical Report) and a companion video‑preview paper.

---

Who’s Behind It?

Developed by the OpenMOSS team, built on NVIDIA’s Megatron‑LM, Qwen language models, and the SGLang serving framework. The project is actively maintained (latest news from Sep 2026) and released under an open‑source license.

---

Quick Takeaway

• What – 11 B video‑language model family for real‑time and offline video understanding.
• Why it matters – First open‑source model that can watch a live stream and talk back instantly, with built‑in silence and correction capabilities.
• How to try – Install dependencies, run the realtime script or use the provided FastAPI/WebSocket service; quantized checkpoints let you do it on a single 24 GB GPU.
• Next steps – Look at the SGLang‑Omni backend for multi‑user serving, or fine‑tune on your own video data using the LoRA scripts.