As artificial intelligence systems become increasingly capable of understanding videos, a new study highlights the challenges they face when trying to keep up with fast-paced, real-world scenes. Researchers have introduced FastBench, a novel benchmark designed to test how well video-based AI models — known as Streaming Video Large Language Models (VLMs) — can perceive and interpret rapidly changing video streams. This matters because many practical applications, from surveillance to autonomous driving, require AI to process high-speed events accurately and continuously.
Key Takeaways
- Existing video AI models struggle to handle high-dynamic scenarios where events happen quickly and continuously.
- FastBench offers a new way to evaluate AI perception using 306 carefully curated question-and-answer pairs across eight different domains.
- Increasing the frame sampling rate (how many video frames the AI analyzes per second) improves performance but only up to a point, as models must balance memory limits and processing speed.
- A new method called ProactiveFrame, which dynamically adjusts frame rates based on text cues, outperforms traditional uniform sampling but still falls short of ideal performance.
Traditional benchmarks for video understanding have largely focused on slow-moving or low-dynamic scenes, where changes happen gradually and can be captured by sampling just a few frames per second. However, in many real-world situations—like sports, traffic monitoring, or emergency response—important events unfold quickly and require AI to maintain a fine-grained, up-to-date understanding of the scene. The researchers behind FastBench recognized this gap and developed a dataset and evaluation pipeline that reflect these high-speed, complex scenarios.
FastBench’s approach involves creating realistic video streams with high frame rates and generating question-answer pairs grounded in specific trajectories, or paths of objects moving through the video. To ensure quality and relevance, the questions are filtered to include only those that can be answered at a lower frame rate baseline (2 frames per second), then verified with advanced tracking tools and multiple rounds of human review. The dataset spans various domains and tests multiple capabilities of AI models, including understanding events that happened in the past, are happening now, or will happen soon.
One key challenge for streaming video AI is the limited “context budget”—the amount of past information the model can remember and process at once. Models must decide how to allocate this budget between looking at recent frames in high detail and keeping a broader but sparser history. To tackle this, the team introduced ProactiveFrame, a training-free technique that uses textual signals to dynamically adjust which frames to analyze at high resolution. This method maintains a “dual-tier sliding window” that keeps recent frames densely sampled while compressing older frames into a more sparse history.
In their experiments, the researchers found that even the best-performing model tested, Gemini-3.5-Flash, achieved only about 50% accuracy on FastBench’s questions. Another model, Qwen3-VL-8B, improved its accuracy from 32.9% at 2 FPS to 44.6% at 24 FPS, showing that denser sampling helps but with diminishing returns. ProactiveFrame improved performance modestly compared to uniform sampling but still lagged behind an “oracle” approach that perfectly knows when to focus on finer temporal details. This suggests current AI systems struggle to autonomously recognize when rapid changes require closer attention.
FastBench opens up new opportunities for researchers to develop and test video AI models that can better handle the demands of real-time, high-dynamic environments. Improving these capabilities could enhance applications like autonomous vehicles, security monitoring, and live event analysis, where missing a fast-moving detail can have significant consequences. Future work may focus on smarter frame selection strategies and more efficient memory use to help AI keep pace with the world as it unfolds.
Based on research published on arXiv by Yuxuan Hu, Weikang Shi, Yang Bo et al..