Core Interaction Paradigm
To equip the model with the ability to handle real-world pacing like a human, we discarded traditional per-second polling and external schedulers, directly instilling MOSS-VL-Realtime with three core behaviors:
Breaking the latency of "watch first, answer later".Users can ask at any second, and the model instantly responds based on the current frame.
"letting the bullet fly".Independently remaining silent and waiting when information is insufficient or no key events occur.
Cognition updates dynamically with the scene.Instantly adjusting and correcting previous answers as new frames flow in.
These three natural interactions stem from a single underlying design: video frames, user questions, and model answers are aligned into a single token stream. Dialogue occurs amidst continuously flowing frames, and questions act like "live chat", seamlessly inserted at any point in the timeline.
Figure 1 illustrates how this unified and interleaved streaming input sequence operates under the hood.
Figure 1: Streaming input representation mechanism.Top (Column A) represents human intuitive perception: watching and answering are parallel, instantly correcting when scenes change. Bottom (Column B) represents actual sequence representation: the interaction is encoded as an interleaved token sequence, embedding absolute-timestamped frames into open turns. Questions append anytime, and <|silence|> acts as an explicit silence token in decision-making.
Architecture & Spatiotemporal Representation
The architecture of MOSS-VL-Realtime is specifically designed for long-context streaming video understanding. Core components include:
- Decoupled Cross-Attention: Decouples cross-attention computation between video feature extraction and text generation, significantly reducing end-to-end throughput and latency overhead on high-frequency frames.
- Absolute Timestamps: Encodes absolute time info for each frame of the input video sequence, allowing the model to perceive physical flow, independent of fixed FPS or relative position.
- Cross-dimensional Rotary Positional Encoding (XRoPE): Constructs a unified spatiotemporal representation, uniformly mapping the spatial dimensions $(h, w)$ and temporal dimension $t$ of video features, providing precise spatiotemporal localization in long-context interactions.
This interaction and spatiotemporal modeling capability stems not only from a rational architecture but from a comprehensive rebuild of multimodal corpora. We introduced a new streaming SFT interaction training paradigm, enabling the model to precisely predict when to speak or proactively stay silent at any moment of sequence generation.
Capability Demos
Monitoring & Alerts
DEMO 01Snowy street surveillance. Maintains realistic silence when nothing happens, and instantly alerts the moment the target person falls.
DEMO 02Watches for one agreed-upon event only: the moment the cat touches the carrot it says “nice!”, and stays silent the rest of the time.
Live Commentary & Presentation
DEMO 03Live broadcast a football match with professional TV commentary tone; passes, shots, and goals are captured and called instantly.
DEMO 04Delivers an impromptu talk over a slide deck, picking up each page the moment it appears and carrying the talk from opening to close.
Time & Number Sense
DEMO 05Catches every successful shot and calls out the running count in real time.
DEMO 06Observe the pea growth timelapse, capture key changes, and accurately report the day they occurred.
DEMO 07Watches a hand solving the problem and transcribes each new line of formula as soon as it is written.
Interactive Reading & Feedback
DEMO 08Plays a palm-reading master: waits until the palm is clearly visible, then reads the lines one by one while observing.
DEMO 09Comments on the outfit in front of the camera; when the user returns in new clothes, it notices the change at once and continues with advice for the new look.
Model Performance
MOSS-VL-Realtime demonstrates significantly stronger streaming interaction, reaching open-source SOTA on streaming video understanding benchmarks while delivering more reliable, real-time responses on complex dynamic tasks.
At the core interaction level, the model stays in sync with the physical world through industry-leading proactive speaking — independently and precisely deciding when to speak and when to stay silent.
Meanwhile, its streaming perception, spatiotemporal grounding, and long-context capabilities keep strengthening: facing sudden events and scene reversals, the model always corrects itself instantly.
Built on these breakthroughs, the model achieves open-source SOTA results across multiple streaming benchmarks, keeps pushing the boundary of seamless streaming interaction, and understands temporal changes, key actions, and sudden events with unprecedented sharpness.
Streaming VLM must not only understand "content" but precisely judge "timing". To fully evaluate MOSS-VL-Realtime's real-time inference capabilities, we conducted an in-depth evaluation on four core benchmarks. The figure below breaks every benchmark down subset by subset for a full side-by-side comparison.
Offline Foundation
During this iteration, we systematically refactored and deeply optimized the data system, solidifying a powerful foundation of fundamental capabilities. Thanks to this, the co-generation Instruct model not only maintains a highly robust benchmark profile in offline capabilities, but also achieves significant improvements in fine-grained multimodal perception (V*Bench 89.0, BLINK 78.0) and high-difficulty temporal action grounding tasks. It continues to deliver remarkably stable performance in long-context video understanding (VideoMME, MLVU) and complex document and chart analysis (DocVQA, ChartQA), comprehensively aligning with or even leading top-tier open-source baselines of comparable scale.
Citation
@misc{mossvlrealtime2026,
title = {MOSS-VL-Realtime},
author = {OpenMOSS Team},
year = {2026}
}