MOSS-VL: Open-Weight Vision-Language Models for Real-Time Video Streams

Moving from “watch first, answer later” to continuous perception, interruption, proactive silence, and dynamic correction.

Short answer: MOSS-VL is OpenMOSS’s open-weight vision-language model family. MOSS-VL-Realtime is designed for continuous video streams: it consumes timestamped frames incrementally, accepts questions at any moment, can stay silent when context is insufficient, and updates an earlier answer when the scene changes.

MOSS-VL-Realtime

Real-time interaction over cameras, screens, live streams, and continuous video sources.

MOSS-VL-Instruct-0708

Offline image/video understanding, long-video analysis, and deep dialogue.

MOSS-VL-Base-0708

An open foundation for continued pre-training, SFT, and LoRA adaptation.

Shared scale

All three current models are 11B with a 256K context and are available through Hugging Face and ModelScope.

What makes real-time video different?

Offline video QA can wait for the full recording. A camera assistant, robot, screen monitor, or live production tool cannot. MOSS-VL-Realtime treats the video timeline as an active input: visual perception and text generation run together, new questions can arrive while frames continue to stream, and the model can decide whether an event deserves a response.

Architecture in three ideas

Cross-Attention separates visual encoding from language reasoning for dynamic streams and interleaved image/video/text input. Absolute timestamps anchor every sampled frame to a precise point in time. XRoPE maps text tokens and video patches into a shared (t, h, w) space for fine-grained temporal and spatial grounding.

The project README reports leading results on the proactive evaluations of OmniMMI, StreamingBench, and ProactiveVideoQA. Those claims should be read together with the official technical blog and paper; the comparison below is a positioning guide, not a single cross-model leaderboard.

How does MOSS-VL compare with well-known models?

Model / family	Primary positioning	Continuous stream	Weights and fine-tuning	Choose it when…
MOSS-VL	Real-time video + offline long video + open base	Native goal: incremental frames, proactive silence, dynamic correction	Open weights; Base, SFT, LoRA, and SGLang paths	You want an open model built around “watch while answering”
GPT-4o	Closed general-purpose multimodal assistant	Depends on product/API integration	Hosted API; no downloadable weights	You prefer a managed general multimodal service
Gemini 2.5 Pro	Closed long-context multimodal model	Service-defined multimodal input and deployment boundary	Hosted API; no downloadable weights	You need a managed long-context model
Qwen2.5-VL	Open image/video understanding and VQA	Streaming requires application-level scheduling	Open weights; local deployment and fine-tuning	You need a strong open offline VLM baseline
InternVL3	Open large-scale vision-language family	Usually needs application-level stream/state management	Open research ecosystem	You care about high-resolution VLM research
LLaVA-OneVision	Open vision-instruction community baseline	Sampling, caching, and scheduling live in the application	Open ecosystem for experiments	You need a widely used vision-instruction baseline

In other words, GPT-4o and Gemini 2.5 Pro are hosted reference points; Qwen2.5-VL, InternVL3, and LLaVA-OneVision are open VLM reference points; MOSS-VL’s distinctive bet is to make continuous video interaction a first-class model objective while keeping offline and fine-tuning variants in the same family.

Quick start

conda create -n moss_vl python=3.12 pip -y
conda activate moss_vl
pip install -i https://pypi.org/simple --no-build-isolation -r requirements.txt

CUDA_VISIBLE_DEVICES=0 python realtime_inference/run_online_inference.py \
  --checkpoint OpenMOSS-Team/MOSS-VL-Realtime \
  --source video --video path/to/example.mp4 \
  --sample-fps 1 --playback-speed 1 --max-frames 256

Use model.create_realtime_session(...) for direct frame/prompt/output control, model.online_generate(...) for queue-based workers, or --serve for a FastAPI WebSocket service.

Resources

GitHub: OpenMOSS/MOSS-VL · Hugging Face · ModelScope
Paper · Technical blog · Apache-2.0 License
Reference projects: Qwen2.5-VL, InternVL, LLaVA-OneVision, GPT-4o, and Gemini.

Star OpenMOSS/MOSS-VL if you are building or studying real-time video understanding.

Note

The model facts and performance language follow the official OpenMOSS repository, model pages, technical blog, and paper. The model table compares product and engineering positioning; it is not a unified benchmark ranking.