---
title: Streaming video understanding goes real-time: FrameBlend cuts prefill 20%, anomaly detection fits under 1 GB
description: FrameBlend's selective KV recomputation trims per-frame prefill FLOPs by over 20% and processing time by 14-59% while improving accuracy. A new survey maps how agents should actively sample video instead of watching it all, and MS2CAST runs anomaly detection with peak GPU memory under 1 GB.
canonical: https://hangingcontext.com/stories/2026-10-22/streaming-video-real-time/
last-updated: 2026-10-22
url: https://hangingcontext.com/stories/2026-10-22/streaming-video-real-time/
published: 2026-10-22T12:45:43Z
edition: 2026-10-22T12
newsroom: artificial-intelligence
beat: research
trust: reported
source_count: 3
cited_manifest_ids: ['1792666336936766938', '1792664915278681649', '1792664915278074169']
---

# Streaming video understanding goes real-time: FrameBlend cuts prefill 20%, anomaly detection fits under 1 GB

*FrameBlend's selective KV recomputation trims per-frame prefill FLOPs by over 20% and processing time by 14-59% while improving accuracy. A new survey maps how agents should actively sample video instead of watching it all, and MS2CAST runs anomaly detection with peak GPU memory under 1 GB.*

Reported · 3 sources · Artificial Intelligence · Published 2026-10-22 12:45 UTC

## Key facts

- FrameBlend reports that selective KV recomputation at query time cuts per-frame prefill FLOPs by over 20% and total processing time by 14-59% for retrieval-based streaming video frameworks, while improving accuracy by up to 3.9 percentage points across five streaming and offline benchmarks. (self-reported) [cites: 1792666336936766938]
- A survey preprint introduces a taxonomy for agentic video understanding, mapping bottlenecks like context limits and evidence sparsity to design patterns: adaptive state construction, active evidence acquisition, and role-specialized coordination. (self-reported) [cites: 1792664915278681649]
- MS2CAST, a video anomaly detection framework, maintains peak GPU memory usage under 1 GB while achieving competitive ROC AUC scores on the UCSD Ped2, CUHK Avenue and ShanghaiTech datasets, replacing optical flow estimation with SwinLSTM or Vision Mamba blocks. (self-reported) [cites: 1792664915278074169]

## Don't recompute what you don't need

Streaming video breaks the normal LLM serving contract. Frames arrive continuously, the attention window gets fixed during idle-time prefill, and by the time a query arrives the cached context no longer matches what the question needs. FrameBlend, in a preprint, fixes the mismatch at query time with selective KV recomputation: keep the short idle-time attention windows that make prefill cheap, then recompute only the KV entries the query actually needs. The authors report per-frame prefill FLOPs down by over 20%, total processing time down 14 to 59% for retrieval-based streaming video understanding frameworks, and accuracy up by as much as 3.9 percentage points across five streaming and offline benchmarks. [cites: 1792666336936766938]

Note the direction of the accuracy number: this is not a trade where cheaper means dumber. Recomputing the right entries at query time also beat the standard frameworks on accuracy, which suggests the idle-time cache was not just expensive but wrong. That is the authors' reading of their own results, and it is the result most worth reproducing before building on it. [cites: 1792666336936766938]

## The agent should choose what to watch

A survey preprint zooms out and asks what the architecture should look like when video understanding stops being passive inference and becomes an agent's job. The survey's taxonomy maps the field's bottlenecks to design patterns: where long-form video creates context bottlenecks, build state adaptively as the stream arrives; where evidence is sparse, acquire it actively instead of processing every frame; where modalities disagree, coordinate specialized roles to resolve the ambiguity. In this framing, the fix for FrameBlend-style context problems is structural: agents that decide what to remember and what to look at next, rather than pipelines that watch everything. [cites: 1792664915278681649]

> **From Synorb** — Claims with a source link. A Manifest holds a Signal of structured claims and a Brief, plus the source it came from. Your agent can check a sentence against it. [Read the guide](https://synorb.com/connect/start)


The taxonomy is a design vocabulary more than a verdict. It gives builders a way to audit a video-agent architecture for gaps in evidence memory and temporal tracking, which is useful precisely because the field does not yet agree on how to measure these systems. [cites: 1792664915278681649]

## Anomaly detection on a gigabyte

The third paper shows how far efficiency engineering has come on the detection side. MS2CAST is a video anomaly detection framework that drops optical flow estimation, traditionally the expensive step, and replaces it with interchangeable spatiotemporal modules, SwinLSTM or Vision Mamba blocks operating in the latent space, plus a multi-scale spatial cross-attention encoder. The headline constraint is peak GPU memory under 1 GB, with competitive ROC AUC scores on the UCSD Ped2, CUHK Avenue and ShanghaiTech datasets. [cites: 1792664915278074169]

Sub-gigabyte anomaly detection is a deployment story, not a leaderboard story. It puts always-on video monitoring within reach of edge hardware and small cloud instances where optical-flow pipelines never fit, which is where most real surveillance and monitoring budgets actually live. [cites: 1792664915278074169]

## What this adds up to

Read together, the three preprints describe video AI growing out of the batch-processing era. FrameBlend makes streaming inference cheaper and more accurate at the same time, the survey gives builders a design vocabulary for agents that sample video actively, and MS2CAST shows the detection workload can live on modest hardware. The honest caveat, flagged by the survey authors themselves, is measurement: without standardized benchmarks for agentic and streaming video, every efficiency claim in this space is still self-graded. [cites: 1792666336936766938, 1792664915278681649, 1792664915278074169]

## What is still unresolved

- The survey authors note a lack of standardized benchmarks specifically for agentic-native temporal modeling.
- FrameBlend's evaluation is the authors' own, and no code or dataset release was mentioned.
- MS2CAST reports competitive ROC AUC scores without publishing numeric scores in the available evidence.

## Decision card

- **Useful for:** Developers building real-time video-LLM pipelines or edge surveillance systems; Teams auditing video-agent architectures against a design taxonomy
- **Not for:** Readers looking for peer-reviewed results; all three papers are preprints
- **Compare against:** Standard offline video-LLM frameworks and optical-flow-based anomaly detectors

## Provenance ledger

| Manifest id | Title | Source | Claims used | Published |
|---|---|---|---|---|
| `1792666336936766938` | Cache Locally, Recompute Selectively for Streaming Video Understanding | [arxiv-model-efficiency-engineering](https://arxiv.org/pdf/2610.29229v1) | 1792666611065173036, 1792666611067892526, 1792666611069737278 | 10-22 00:00 |
| `1792664915278681649` | Agentic Video Understanding: A Survey | [arxiv-multimodal-document-ai](https://arxiv.org/pdf/2609.31713v2) | 1792667227418156764, 1792667227420656158, 1792667227422432934 | 10-22 00:00 |
| `1792664915278074169` | MS2CAST: A Video Anomaly Detection Framework via Multi-Scale Spatial Cross-Attention and Efficient Spatiotemporal Networks | [arxiv-multimodal-document-ai](https://arxiv.org/pdf/2610.29649v1) | 1792665188831835992, 1792665188839194747, 1792665188842112509 | 10-22 00:00 |

## Run these Manifests yourself

Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.

1. **Get a Synorb Key.** Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{"client":"your-agent-name"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; 3 of them are starter Beacons.
2. **Read each cited Manifest by id.** REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief.
3. **Check the claim.** Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them.

Cited Manifest ids:

- `1792666336936766938`
- `1792664915278681649`
- `1792664915278074169`

```sh
for id in 1792666336936766938 1792664915278681649 1792664915278074169; do
  curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id"
done
```

> **From Synorb** — Ask what Synorb covers. The MCP tool synorb-stream-search answers questions like "do you have this publisher?" without using any Manifest allowance. [Connect and ask](https://synorb.com/connect/start)

## Related stories

- [The serving stack's memory wars: unified KV addressing hits 2.51x, GPU-NPU disaggregation cuts cost 1.72x](https://hangingcontext.com/stories/2026-10-22/serving-stack-memory-wars/)
- [Scaling laws get sharper: MoE capacity theory, VLM transfer prediction, and a 20k-parameter jet tagger](https://hangingcontext.com/stories/2026-10-22/scaling-laws-three-ways/)
- [Robot navigation's pretraining playbook: train smarter, transfer further, simulate once](https://hangingcontext.com/stories/2026-10-22/robot-navigation-pretraining-playbook/)
- [Reasoning RL gets surgical: prefix-level trust regions, retry-aware credit, and risk-sensitive GRPO for diffusion models](https://hangingcontext.com/stories/2026-10-22/reasoning-rl-surgical/)

## Links

- HTML: https://hangingcontext.com/stories/2026-10-22/streaming-video-real-time/
- JSON: https://hangingcontext.com/stories/2026-10-22/streaming-video-real-time/index.json
- Markdown: https://hangingcontext.com/stories/2026-10-22/streaming-video-real-time/index.md

Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.

---
Reuse (terms v1): Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product. Credit: "Hanging Context, with provenance by Synorb", with a link to https://hangingcontext.com/stories/2026-10-22/streaming-video-real-time/. Terms: https://hangingcontext.com/terms/reuse/