{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "482",
  "slug": "llms-fail-like-humans",
  "edition": "2026-10-23T00",
  "url": "https://hangingcontext.com/stories/2026-10-23/llms-fail-like-humans/",
  "newsroom": "artificial-intelligence",
  "beat": "research",
  "newsroom_title": "Artificial Intelligence",
  "angle": "LLMs fail the way humans fail: framing effects, a broken sense of time, and reasoning that diverges across languages.",
  "headline": "LLMs fail like humans: framing effects, a broken sense of time, divergent reasoning across languages",
  "dek": "Three preprints this week probe the same uncomfortable pattern: models that change their advice when the wording changes, memorize time instead of reasoning about it, and reason differently in each language.",
  "published_at": "2026-10-23T02:01:20Z",
  "labels": {
    "trust": "reported",
    "trust_text": "Reported · 3 sources",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 3,
  "publishers": [
    "arxiv-ai-security-privacy-safety",
    "arxiv-llm-evals-benchmarks"
  ],
  "key_facts": [
    {
      "text": "In a preprint, the authors report that gain-loss framing changed the recommendation category in 32.4% of LLM decision-support groups; probability canonicalization reduced the sensitivity.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792675125246656057",
          "claim_ids": [
            "1792675430239939506",
            "1792675430241069175"
          ]
        }
      ]
    },
    {
      "text": "In a preprint on temporal reasoning, fine-tuning on temporal arithmetic or semantic subsets produced only dataset-specific gains of 8 to 11 percentage points, with cross-dataset transfer of minus 3.2 to 1 percentage points.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792675125246709170",
          "claim_ids": [
            "1792675438000860444",
            "1792675438003164461"
          ]
        }
      ]
    },
    {
      "text": "The same preprint reports that knowledge, not quantitative or logical reasoning, is the primary driver of performance on current temporal reasoning benchmarks.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792675125246709170",
          "claim_ids": [
            "1792675438005697264"
          ]
        }
      ]
    },
    {
      "text": "In a preprint on multilingual reasoning, the authors report that standard holistic similarity metrics systematically overstate reasoning consistency, and that 52.1% to 67.3% of top-quartile holistic matches fell outside the top quartile of structural similarity.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792672392463426104",
          "claim_ids": [
            "1792672671007903313",
            "1792672671008661370"
          ]
        }
      ]
    },
    {
      "text": "The multilingual preprint reports that human judgments of reasoning consistency aligned more closely with structural similarity than with holistic baseline similarity.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792672392463426104",
          "claim_ids": [
            "1792672671010309515"
          ]
        }
      ]
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "The framing problem",
      "paragraphs": [
        {
          "markdown": "In a preprint on gain-loss framing in high-stakes decision support, researchers tested whether large language models give consistent advice when identical probabilities are described as gains versus losses. They found that gain-loss framing shifted the recommendation category in 32.4% of decision-support groups. The proposed fix is probability canonicalization, which the authors report reduces the sensitivity.",
          "cites": [
            {
              "manifest_id": "1792675125246656057",
              "claim_ids": [
                "1792675430239939506",
                "1792675430241069175"
              ]
            }
          ]
        },
        {
          "markdown": "This is the oldest bias in behavioral economics wearing a new uniform. Humans famously flip their choices when a sure gain is reframed as an avoided loss, and the preprint suggests models inherited the same inconsistency from their training data. For anyone deploying models as decision support, the implication is blunt: the advice your model gives may depend on how the question was worded, not on the numbers underneath it.",
          "cites": [
            {
              "manifest_id": "1792675125246656057",
              "claim_ids": [
                "1792675430239939506"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Time doesn't transfer",
      "paragraphs": [
        {
          "markdown": "A second preprint on temporal reasoning in large language models asks whether models actually reason about time. Fine-tuning on temporal arithmetic or semantic subsets produced dataset-specific gains of 8 to 11 percentage points, but the skills transferred between datasets in the same mode at rates of minus 3.2 to 1 percentage points. The improvement stayed where it was taught.",
          "cites": [
            {
              "manifest_id": "1792675125246709170",
              "claim_ids": [
                "1792675438000860444",
                "1792675438003164461"
              ]
            }
          ]
        },
        {
          "markdown": "The authors' sharper claim is that knowledge, not quantitative or logical reasoning, is the primary driver of performance on current temporal benchmarks, and that those benchmarks fail to distinguish arithmetic from semantic modes. Scores on temporal reasoning tests, they argue, mostly measure memorized patterns. That forces a rethink of how much to trust an agent that schedules, plans or forecasts.",
          "cites": [
            {
              "manifest_id": "1792675125246709170",
              "claim_ids": [
                "1792675438005697264"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Same answer, different reasons",
      "paragraphs": [
        {
          "markdown": "A third preprint on trace-level divergence in multilingual large language models takes on multilingual consistency. Standard holistic similarity metrics, such as cosine similarity of embeddings, systematically overstate how consistent a model's reasoning is across languages, the authors report. Among examples in the top quartile of holistic similarity, 52.1% to 67.3% fell outside the top quartile of structural similarity: the answers matched while the reasoning traces diverged.",
          "cites": [
            {
              "manifest_id": "1792672392463426104",
              "claim_ids": [
                "1792672671007903313",
                "1792672671008661370"
              ]
            }
          ]
        },
        {
          "markdown": "Human judges, asked to rate reasoning consistency, agreed more with the structural measure than the holistic baseline. The practical reading is uncomfortable: shipping the same model in two languages can mean shipping two different reasoners, with the divergence invisible to the metrics most teams use.",
          "cites": [
            {
              "manifest_id": "1792672392463426104",
              "claim_ids": [
                "1792672671010309515"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "What changes for professionals",
      "paragraphs": [
        {
          "markdown": "None of the three papers claims models are useless, and all three are preprints, so their results should be treated as the authors' own findings rather than settled science. But together they redraw what testing should mean. Canonicalize probabilities before asking for advice. Treat temporal benchmark scores as memorization scores until proven otherwise. And check reasoning traces, not just final answers, when a model operates in more than one language.",
          "cites": [
            {
              "manifest_id": "1792675125246656057",
              "claim_ids": [
                "1792675430241069175"
              ]
            },
            {
              "manifest_id": "1792675125246709170",
              "claim_ids": [
                "1792675438005697264"
              ]
            },
            {
              "manifest_id": "1792672392463426104",
              "claim_ids": [
                "1792672671008661370"
              ]
            }
          ]
        },
        {
          "markdown": "The deeper pattern is that models fail the way their training data fails. Humans frame, humans memorize dates, humans reason differently in a second language. Grading a model only on its outputs is like grading a student on the answer key. All three preprints point, in different ways, at the same audit target: the process, not the product.",
          "cites": [
            {
              "manifest_id": "1792675125246656057",
              "claim_ids": [
                "1792675430239939506"
              ]
            },
            {
              "manifest_id": "1792675125246709170",
              "claim_ids": [
                "1792675438000860444"
              ]
            },
            {
              "manifest_id": "1792672392463426104",
              "claim_ids": [
                "1792672671007903313"
              ]
            }
          ]
        },
        {
          "markdown": "The throughline for a professional reader is liability. A model that changes its recommendation because a probability was phrased as a loss, or that answers confidently in a language whose reasoning it cannot show its work in, is a model whose outputs need a human-grade audit trail. The three preprints, taken together, suggest the audit should start before the output: canonicalize the inputs, distrust the benchmark, and read the trace.",
          "cites": [
            {
              "manifest_id": "1792675125246656057",
              "claim_ids": [
                "1792675430241069175"
              ]
            },
            {
              "manifest_id": "1792675125246709170",
              "claim_ids": [
                "1792675438005697264"
              ]
            },
            {
              "manifest_id": "1792672392463426104",
              "claim_ids": [
                "1792672671010309515"
              ]
            }
          ]
        }
      ]
    }
  ],
  "unresolved": [
    "Whether probability canonicalization holds up outside the tested decision-support scenarios",
    "What a non-memorization-based temporal reasoning benchmark would look like",
    "Whether trace-level divergence predicts real deployment failures in multilingual products"
  ],
  "decision_card": {
    "useful_for": [
      "Professionals building agents that advise on decisions, schedules or multilingual tasks",
      "Teams designing LLM evaluations beyond output accuracy"
    ],
    "not_for": [
      "Model trainers looking for peer-reviewed, settled results"
    ],
    "compare_against": [
      "Your own red-team results on framing and temporal prompts",
      "Trace-level checks in your multilingual evals"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-10-23T00"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1792675125246656057",
    "1792675125246709170",
    "1792672392463426104"
  ],
  "manifests": [
    {
      "manifest_id": "1792675125246656057",
      "title": "Same Probabilities, Different Advice: Gain-Loss Framing Effects in High-Stakes LLM Decision Support",
      "source_name": "arxiv-llm-evals-benchmarks",
      "source_url": "https://arxiv.org/pdf/2610.29294v1",
      "published_date": "2026-10-22"
    },
    {
      "manifest_id": "1792675125246709170",
      "title": "Keeping Time vs. Understanding Time: Neither Alone Explains Temporal Reasoning in LLMs",
      "source_name": "arxiv-llm-evals-benchmarks",
      "source_url": "https://arxiv.org/pdf/2610.29507v1",
      "published_date": "2026-10-22"
    },
    {
      "manifest_id": "1792672392463426104",
      "title": "Same Answer, Different Reasons: Trace-Level Divergence in Multilingual Large Language Models",
      "source_name": "arxiv-ai-security-privacy-safety",
      "source_url": "https://arxiv.org/pdf/2610.29092v1",
      "published_date": "2026-10-22"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1792675125246656057",
      "1792675125246709170",
      "1792672392463426104"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; 3 of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1792675125246656057",
      "all": "for id in 1792675125246656057 1792675125246709170 1792672392463426104; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-10-23/llms-fail-like-humans/",
    "json": "https://hangingcontext.com/stories/2026-10-23/llms-fail-like-humans/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-10-23/llms-fail-like-humans/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-10-23T00/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-10-23/legal-ai-moves-to-mandates/",
      "headline": "Law firms stop piloting AI and start rolling it out to everyone",
      "slug": "legal-ai-moves-to-mandates"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-22/brain-foundation-models-eeg/",
      "headline": "The brain gets foundation models: 97% fewer parameters, montage-proof pretraining, and a stress test for missing sensors",
      "slug": "brain-foundation-models-eeg"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-22/clinical-ai-vlm-mirage/",
      "headline": "Clinical AI grows up: automated heart imaging, expert-level pathogen calls, a foot-temperature warning",
      "slug": "clinical-ai-vlm-mirage"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Save what you follow as a Beacon",
    "text": "A Beacon is a saved query that your agent pulls with an explicit lookback window. Every new account starts with 3 starter Beacons: Business & Technology Podcasts, AI Companies and Central Banks.",
    "cta": {
      "label": "See Beacons",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-10-23/llms-fail-like-humans/"
  }
}