More thinking buys confidence, not better answers
In a preprint, extra reasoning effort raised shopping agents' confidence without improving their choices. Two more preprints show the fix is to think less, and only where it counts.
- In a preprint, ten frontier models were benchmarked on three product-choice platforms. Increasing thinking effort does not increase decision quality in a statistically meaningful way, but it raises stated confidence at a statistically significant level. Self-reported m:…3393
- In a preprint, AnyS1 uses a 4B parameter pointer model for routine decisions and reduces episode costs by 75-91% and model-call time by 63-92% relative to an uncompiled System-2 agent. Self-reported m:…5058
- In a preprint, DR-BOLT achieves a 2.90-fold end-to-end training speedup over reproduced GRPO and maintains a 24.1% cross-benchmark slow ratio. Self-reported m:…7409
- In a preprint, certified policies can skip helper messages in 59-80% of cases while retaining 76-95% of performance gains. Self-reported m:…9773
Confident, wrong, and wrong together
The most useful finding in this week's reasoning research is a negative one. In a preprint, increasing thinking effort in LLM shopping agents does not increase decision quality in a statistically meaningful way, but it raises stated confidence at a statistically significant level. The models sound surer. They are not righter. m:…3393
The study is broad enough to take seriously. Ten frontier models were benchmarked on three product-choice platforms, and most models are not well-calibrated at every effort level. Errors are strongly shared across models, and when several models err, a large number of them pick the same wrong product. LLMs are increasingly deployed as autonomous shopping agents, which makes the combination, confident, wrong, and wrong in the same way, the thing to design around. m:…3393
Rationing the slow thinking
Two more preprints, both about spending thought more carefully, point the same way. In a preprint, a system called AnyS1 uses a 4B parameter pointer model as a System-1 component for routine decisions, falling back to a large System-2 model when confidence falls below a calibrated threshold. AnyS1 reduces episode costs by 75-91% relative to an uncompiled System-2 agent and reduces model-call time by 63-92%, while maintaining task reliability across three unseen benchmarks. It was trained using grounded enumerable decisions mined across over 1,000 public skills. m:…5058
In another preprint, a method called DR-BOLT optimizes vision-language model inference by routing inputs between fast and slow reasoning paths. DR-BOLT achieves a 2.90-fold end-to-end training speedup over reproduced GRPO and maintains a 24.1% cross-benchmark slow ratio, balancing reasoning depth with inference cost. It improves score-length trade-offs over evaluated offline and online baselines on ScienceQA and held-out transfer benchmarks. The design treats slow thinking as a scarce input to be rationed, not a default. m:…7409
A fourth preprint applies the same rationing to teams of agents. Certified policies can skip helper messages in 59-80% of cases while retaining 76-95% of performance gains, and message skipping effectiveness is determined by the predictability of answer alteration, not the helper agent's accuracy. Oracle headroom is a weak indicator of message utility, as control inputs reproduce 76% of the oracle's gain. The message is that most of what agents say to each other adds little, and the skill is knowing which messages matter before they are sent. m:…9773
Prove the reasoning earned its keep
The calibration problem is the thread that ties the four together. Most models are not well-calibrated at every effort level, so a confidence score cannot be read as a probability that the answer is right. AnyS1 and DR-BOLT both work around that by using a calibrated threshold or a fixed slow ratio rather than trusting the model's own sense of certainty. The shopping-agent study is the warning about what happens when a system does trust it: confidence rises, accuracy does not, and the errors cluster on the same wrong product. m:…3393 m:…5058 m:…7409
Taken together, the four preprints describe a single discipline: spend reasoning where it changes the answer, and stop paying for confidence. Shopping agents that think harder get more confident without getting more accurate. Systems that route easy decisions to a small fast model, or skip low-value messages entirely, keep most of the performance at a fraction of the cost. For builders, the question is no longer how much reasoning to buy. It is how to prove a given unit of reasoning earned its keep. m:…3393 m:…5058
What is still unresolved
- Whether the confidence-without-accuracy result holds for shopping agents choosing among products with larger price gaps.
- Whether AnyS1's cost savings hold outside the three unseen benchmarks it was tested on.
- How DR-BOLT's fixed slow ratio should be set for tasks where the cost of a wrong answer is high.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.