OpenMOSS/MOSS-VL

📽️ MOSS‑VL – Open‑weight real‑time video‑language models

What it is – A family of 11 B‑parameter multimodal models that can understand and converse about continuous video streams in real time. The project ships three variants:

Model	Intended use	Context length	Where to get
MOSS‑VL‑Realtime	Interactive, interrupt‑driven dialogue while a video is playing	256 K tokens	🤗 HF / 🤖 ModelScope
MOSS‑VL‑Instruct‑0708	Offline, long‑video Q&A, chat‑style tasks	256 K	🤗 HF
MOSS‑VL‑Base‑0708	Foundation model for further pre‑training or fine‑tuning	256 K	🤗 HF

Core ideas

• Cross‑attention architecture – visual encoder and language decoder are decoupled; a cross‑attention rotary position embedding (XRoPE) maps video patches and text tokens into a unified 3‑D (time, height, width) space, enabling fine‑grained temporal grounding.
• Absolute timestamps – each sampled frame carries a special token with its exact time, so the model can reason about pacing, motion and variable frame‑rates.
• Interruptible streaming – the model processes incoming frames and generates text in parallel. Users may ask a question at any moment; the model can answer immediately, stay silent when information is insufficient, and revise earlier outputs as new frames arrive.
• Proactive silence & correction – built‑in decision logic decides when to speak and when to keep watching, and it can update its answer on‑the‑fly when the visual context changes.

Performance highlights (as reported in the paper & benchmark tables)

• Streaming benchmarks – state‑of‑the‑art open‑source results on OmniMMI, StreamingBench and ProactiveVideoQA, with a top PA@OmniMMI score of 66.0.
• Offline multimodal benchmarks – competitive scores across standard video‑language tasks (details in the technical blog).
• Latency – thanks to the cross‑attention design and a custom FlashAttention‑3 backend, the realtime model runs at open‑source SOTA latency on a single 24 GB GPU (quantized FP8/NF4 checkpoints are provided).

Getting started

1. Environment – Python 3.12, Conda, then pip install -r requirements.txt (the repo bundles a FlashAttention‑3 implementation).
2. Realtime inference – run realtime_inference/run_online_inference.py with a video source, e.g.:

   CUDA_VISIBLE_DEVICES=0 python realtime_inference/run_online_inference.py \
     --checkpoint OpenMOSS-Team/MOSS-VL-Realtime \
     --source video --video path/to/example.mp4 \
     --sample-fps 1 --playback-speed 1 --max-frames 256
   

   Three integration levels are offered: a low‑level session API, a queue‑based online_generate, and a FastAPI WebSocket server (--serve).
3. Offline inference – load with HuggingFace AutoModelForCausalLM + AutoProcessor and call model.offline_batch_generate for interleaved image/video/text queries.
4. Fine‑tuning – scripts under finetune/ support full‑parameter training (vision encoder frozen by default) and LoRA via the peft library.
5. Deployment – pre‑built support for the high‑throughput SGLang serving framework (sglang/ directory) and integration with ModelScope’s ms‑swift CLI.

Ecosystem & community

• Model hubs – checkpoints hosted on Hugging Face and ModelScope, with quantized FP8/NF4 variants (~24 GiB) for single‑GPU inference.
• Tooling – flash‑attention‑3 backend, SGLang serving, ms‑swift inference/fine‑tuning, and LlamaFactory recipes.
• Community – Discord, Feishu group, and a dedicated HF Space for interactive demos.
• Paper & technical report – arXiv [2608.15045] (MOSS‑VL Technical Report) and a preceding pre‑print on the cross‑attention design.

License & citation

• The code is released under the repository’s LICENSE (see LICENSE).
• Cite the technical report and the earlier “MOSS‑Video‑Preview” pre‑print (both provided in the README).

---

Bottom line – MOSS‑VL is a genuine open‑weight video‑language system that pushes the frontier from watch‑then‑answer to watch‑and‑talk in real time, with a research‑grade implementation, ready‑to‑use checkpoints, and a growing ecosystem of inference and fine‑tuning tools.