Vision-language models go on a token diet

Seeing is expensive

Visual tokens are the expensive part of a vision-language model: they drive latency, memory, and cost — the bottleneck all three papers target. Three preprints posted this week attack that cost from different directions — pruning tokens at mixed granularity, pruning across both inference phases, and picking video frames by irreplaceability. Bifocal and GIFT are explicitly training-free, and TwinPrune applies its pruning signal at inference: no retraining bill, just cheaper inference. m:…6998 m:…3738 m:…5632

The shared bet across all three is that inference-time pruning is the right kind of cheap. Bifocal and GIFT are training-free plug-ins; TwinPrune's unified signal scores token importance during inference itself — each cuts the tokens flowing through the model. That matters because visual tokens are the computational bottleneck these methods target. And the evaluations are ambitious: Bifocal is tested across three MLLM families and nine model-budget settings, TwinPrune scales from 8B to 38B parameters, and GIFT is evaluated on long-form video benchmarks. m:…6998 m:…3738 m:…5632

Bifocal: keep the fine details

Bifocal is a two-stage, training-free pruning method that mixes granularities: query-focused evidence plus image-intrinsic evidence, with density-adjusted attention to keep spatial integrity. The headline result, in the authors' preprint: over 90% of unpruned performance retained in every ability group across three MLLM families, using just 25% of the original visual tokens — and first place on average across nine model-budget settings against ten existing pruning methods. m:…6998

The detail that matters is where older methods fail: fine-grained tasks like OCR and visual grounding, which collapse under single-importance rankings and spatial disruption. Bifocal is explicitly built to preserve task-specific evidence there, which points it at resource-constrained deployment rather than just benchmark-chasing. m:…6998

The comparison set is doing real work here: ten existing pruning methods across nine model-budget settings is a broad net, and Bifocal ranking first on average is the kind of result that survives skepticism better than a single-benchmark win. The authors are explicit about who this is for — AI operators and developers optimizing inference latency and cost — not for a leaderboard. The training-free design is the enabler: no gradient steps, no fine-tuning data, just a pruning policy applied at inference. m:…6998

TwinPrune: prune the decode phase too

TwinPrune applies one unified spatial-density signal to both phases of inference: bipartite merging at prefill and KV eviction at decode. Its density top-k eviction beats H2O, SnapKV, and StreamingLLM on TextVQA and MM-Vet in the authors' tests, with compression that improves as models scale from 8B to 38B parameters. m:…3738

The paper's second contribution is methodological and worth repeating: decode-phase pruning claims validated solely on prefill probes are misleading and do not reflect actual decode performance. The authors argue for full-pipeline benchmarks like MM-Vet — a standard the field should probably adopt before trusting anyone's compression numbers. m:…3738

The scale result deserves attention: compression performance improves as models grow from 8B to 38B parameters. And the evaluation warning has teeth beyond this paper. If prefill-only probes flatter decode-phase pruning numbers, then a chunk of the published compression literature may be measuring the wrong thing, and MM-Vet-style full-pipeline benchmarks become the honest yardstick. m:…3738

GIFT: keep only the irreplaceable frames

GIFT, a recent preprint, targets video, replacing uniform frame sampling with an irreplaceability-based selection strategy. Its two ideas: Directed Diversity, which scores a frame's uniqueness conditioned on relevance, and Budget-Aware Refinement, which locks in the highest-irreplaceability frames first and then builds temporal context as the budget allows. m:…5632

The reported payoff: up to 12.5% average improvement across long-form video benchmarks on LLaVA-Video-7B versus uniform sampling. For anyone running video understanding at scale, the practical read is budget control — spend tokens on frames the video cannot do without, and stop paying for the rest. m:…5632

The mechanism is worth unpacking: Directed Diversity scores each frame's uniqueness conditioned on relevance, producing a single irreplaceability score, and Budget-Aware Refinement secures the highest-scoring frames first before spending remaining budget on temporal context. That ordering is the insight — anchor on the frames the video cannot lose, then fill in. At up to 12.5% average improvement on LLaVA-Video-7B over uniform sampling, the question for video pipelines is no longer whether to sample smarter, but how much budget the refinement stage gets. m:…5632