LLMs fail like humans: framing effects, a broken sense of time, divergent reasoning across languages
Three preprints this week probe the same uncomfortable pattern: models that change their advice when the wording changes, memorize time instead of reasoning about it, and reason differently in each language.
- In a preprint, the authors report that gain-loss framing changed the recommendation category in 32.4% of LLM decision-support groups; probability canonicalization reduced the sensitivity. Self-reported m:…6057
- In a preprint on temporal reasoning, fine-tuning on temporal arithmetic or semantic subsets produced only dataset-specific gains of 8 to 11 percentage points, with cross-dataset transfer of minus 3.2 to 1 percentage points. Self-reported m:…9170
- The same preprint reports that knowledge, not quantitative or logical reasoning, is the primary driver of performance on current temporal reasoning benchmarks. Self-reported m:…9170
- In a preprint on multilingual reasoning, the authors report that standard holistic similarity metrics systematically overstate reasoning consistency, and that 52.1% to 67.3% of top-quartile holistic matches fell outside the top quartile of structural similarity. Self-reported m:…6104
- The multilingual preprint reports that human judgments of reasoning consistency aligned more closely with structural similarity than with holistic baseline similarity. Self-reported m:…6104
The framing problem
In a preprint on gain-loss framing in high-stakes decision support, researchers tested whether large language models give consistent advice when identical probabilities are described as gains versus losses. They found that gain-loss framing shifted the recommendation category in 32.4% of decision-support groups. The proposed fix is probability canonicalization, which the authors report reduces the sensitivity. m:…6057
This is the oldest bias in behavioral economics wearing a new uniform. Humans famously flip their choices when a sure gain is reframed as an avoided loss, and the preprint suggests models inherited the same inconsistency from their training data. For anyone deploying models as decision support, the implication is blunt: the advice your model gives may depend on how the question was worded, not on the numbers underneath it. m:…6057
Time doesn't transfer
A second preprint on temporal reasoning in large language models asks whether models actually reason about time. Fine-tuning on temporal arithmetic or semantic subsets produced dataset-specific gains of 8 to 11 percentage points, but the skills transferred between datasets in the same mode at rates of minus 3.2 to 1 percentage points. The improvement stayed where it was taught. m:…9170
The authors' sharper claim is that knowledge, not quantitative or logical reasoning, is the primary driver of performance on current temporal benchmarks, and that those benchmarks fail to distinguish arithmetic from semantic modes. Scores on temporal reasoning tests, they argue, mostly measure memorized patterns. That forces a rethink of how much to trust an agent that schedules, plans or forecasts. m:…9170
Same answer, different reasons
A third preprint on trace-level divergence in multilingual large language models takes on multilingual consistency. Standard holistic similarity metrics, such as cosine similarity of embeddings, systematically overstate how consistent a model's reasoning is across languages, the authors report. Among examples in the top quartile of holistic similarity, 52.1% to 67.3% fell outside the top quartile of structural similarity: the answers matched while the reasoning traces diverged. m:…6104
Human judges, asked to rate reasoning consistency, agreed more with the structural measure than the holistic baseline. The practical reading is uncomfortable: shipping the same model in two languages can mean shipping two different reasoners, with the divergence invisible to the metrics most teams use. m:…6104
What changes for professionals
None of the three papers claims models are useless, and all three are preprints, so their results should be treated as the authors' own findings rather than settled science. But together they redraw what testing should mean. Canonicalize probabilities before asking for advice. Treat temporal benchmark scores as memorization scores until proven otherwise. And check reasoning traces, not just final answers, when a model operates in more than one language. m:…6057 m:…9170 m:…6104
The deeper pattern is that models fail the way their training data fails. Humans frame, humans memorize dates, humans reason differently in a second language. Grading a model only on its outputs is like grading a student on the answer key. All three preprints point, in different ways, at the same audit target: the process, not the product. m:…6057 m:…9170 m:…6104
The throughline for a professional reader is liability. A model that changes its recommendation because a probability was phrased as a loss, or that answers confidently in a language whose reasoning it cannot show its work in, is a model whose outputs need a human-grade audit trail. The three preprints, taken together, suggest the audit should start before the output: canonicalize the inputs, distrust the benchmark, and read the trace. m:…6057 m:…9170 m:…6104
What is still unresolved
- Whether probability canonicalization holds up outside the tested decision-support scenarios
- What a non-memorization-based temporal reasoning benchmark would look like
- Whether trace-level divergence predicts real deployment failures in multilingual products
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.