Hallucination gets a taxonomy: five preprints measure where models invent, drift, and go silent

The span is the unit

Researchers this week introduced SHAM-Bench, a benchmark that aggregates 6,461 examples across six natural-language-generation tasks into a single framework for span-level hallucination detection. The authors present it as an answer to fragmented existing datasets: one yardstick for testing whether detection methods generalize. m:…9661

Their headline result is a trade-off between training regimes. Fine-tuned hallucination detectors perform well in-domain but degrade under distribution shift, while prompt-based methods that incorporate reasoning prove more robust across tasks, the preprint reports. A detector trained to spot bad spans in one task may not survive the next one. m:…9661

The authors are candid about the limits of their own yardstick: span-level evaluation remains challenged by ambiguous span boundaries and noisy labels. Any detector benchmarked on SHAM-Bench inherits that uncertainty, which is worth remembering before treating its rankings as production truth. m:…9661

Completeness versus precision

A separate preprint puts ten large language models through a 70-prompt benchmark of factual reliability and finds Perplexity and DeepSeek V3 achieving the highest overall accuracy. The study frames its contribution as practical: model selection should be tailored to an application's complexity and risk tolerance. m:…7188

The extremes tell the story. The Falcon model achieved zero hallucinations but the lowest recall and highest response deviation, while LLaMA 3's conservative generation method enhanced its resistance to hallucinations, according to the preprint. Saying nothing is one reliable way to never be wrong. m:…7188

The authors describe a fundamental trade-off between response completeness and factual reliability that intensifies as prompts get more complex. For developers, the practical read is that the hallucination question is not which model hallucinates least in the abstract, but which trade-off a given product can afford. m:…7188

Hallucination beyond facts

Hallucination is not always a wrong fact. DialFork, another preprint, names procedural hallucination: a distinct failure mode where conversational agents remain fluent and factually plausible but deviate from the intended conversational process. For task-oriented agents, process compliance is the product. m:…9906

The benchmark forks dialogues and injects controlled errors, producing 36,278 dialogues across 24 tasks and 12 domains for evaluating that drift. Its FlowScore — a dialog-flow-aware metric — outperformed semantic-similarity baselines across seven encoders in detecting procedural deviation, the authors report. m:…9906

Where the error enters the pipeline

A fourth preprint builds a diagnostic framework for text-to-SQL that decomposes complex queries into executable sub-steps, classifying hallucinations as recurrent — present in early steps — or emergent, appearing only at the final stage. Knowing where the logic failed is half of the fix. m:…5040

The results include a wrinkle for prompt engineers: conversational history increases recurrence probabilities while decreasing emergence probabilities. Context grounds the model against late-stage invention but compounds early mistakes — and clause abuse shows higher conditional emergence than other hallucination categories. m:…5040

Finally, a study of 13,919 annotated cases taxonomizes LLM hallucinations in cross-language code translation into three categories — Semantic Consistency Conflicts, Factual Knowledge Conflicts, and Project Context Deviations — and proposes HaluCodeTran, a framework combining BM25-based retrieval with reflective refinement, to mitigate them. Translation invents facts in its own distinctive ways. m:…8048

Taken together, the five preprints push hallucination evaluation from sentence-level impressions toward named, counted failure modes — spans, procedures, trajectories, and translation classes. The through-line is practical: each taxonomy is built so a developer can point at exactly where a system invents. All five, though, are preprints, and their verdicts are only as strong as their test sets. m:…9661 m:…7188 m:…9906 m:…5040 m:…8048