Can the Model See It, or Just Say It? Document Benchmarks Close In

Explainability first

In a preprint, DocXEval provides a systematic framework for evaluating explainability in multimodal document understanding, benchmarking 13 explanation methods across 17 metrics. Occlusion emerged as the most faithful and interpretable method for these tasks. The framework also enables identification of dataset biases and spatial attribution patterns in document models, which means it can show where a model is looking, not only whether its answer was right. m:…7042

That distinction matters because document work is high stakes and spatial. A model that answers correctly for the wrong region, or because of a dataset bias, will fail silently when the layout changes. Faithfulness and complexity metrics showed moderate to strong agreement in these tasks, which suggests the evaluation framework is measuring something stable rather than one metric artifact. m:…7042

Shred the document

A second preprint makes the test harder on purpose. ShredBench tests multimodal models on reconstructing documents from shredded fragments, using an automated pipeline that generates fragmented images from Markdown so that unseen textual sources can be integrated and training data contamination avoided. Current models showed a significant performance gap when moving from intact to shredded documents. m:…2806

The gap widened as fragmentation increased. Moving to 8, 12 and then 16 pieces led to a sharp decline in Normalized Edit Distance for state of the art models. Reconstruction is a stricter test than question answering: a model cannot lean on a plausible summary when it must rebuild the document itself, piece by piece. m:…2806

Edit dense pages precisely

A third preprint tests editing rather than reading. VDE Bench contains 3,506 instruction-guided image editing samples for visual documents, including 1,578 text-region edits and 1,928 table-region edits. Its evaluation protocol combines region-level and full-page measurements to assess textual correctness, structural fidelity and visual content preservation. m:…0638

The authors note that editing dense documents requires precise local changes amid tightly coupled text, tables and layout. That coupling is the whole difficulty. Change one cell and the table, the surrounding text and the page structure all have opinions about the result. A benchmark that measures only the edited region would miss most of the damage. m:…0638

Teach connection, probe corruption

Two more preprints probe perception from opposite sides. One introduces EvidenceConnect, which synthesizes training data by using zoomed views of related regions to generate questions requiring comparison, aggregation or chaining of visual evidence. Its quality control relies on independent model agreement and region-ablation tests to discard questions where the answer does not depend on the visual evidence. The resulting model improved fine-grained and spatial perception, embodied spatial reasoning and multimodal long-context understanding. m:…4360

The other finds a gap between training and inference in vision-language models. Adversarial robustness in autoregressive vision-language models was primarily a property of the language decoder prior, not the visual encoder. The visual encoder corrupted image representations by a comparable margin whether the model produced a correct or a corrupted output. A linear probe on the merger hidden state could separate susceptible and resistant images with an area under the curve of 0.858, which suggests susceptibility is detectable even when the final answer looks fine. m:…2431

Read together, these preprints argue for a higher bar. A document model should explain where it looked, rebuild what was shredded, edit a dense page without breaking its structure, connect evidence across regions, and survive corruption in its visual input. Passing an ordinary accuracy test proves none of those things on its own. m:…7042 m:…2806 m:…0638 m:…4360 m:…2431