Three preprints test what agents can learn in context: exploration, Bayesian optimality, and when not to fine-tune

Teaching exploration without the RL bill

Exploration is the expensive part of reinforcement learning: an agent has to try things, fail, and try again before it learns which actions pay off, and every failed attempt costs interactions with the environment. In a preprint posted this week, researchers describe ASTEROID, an asymmetric on-policy distillation method that teaches exploration in context instead of paying the full RL interaction bill. The reported results are striking: the method achieves about 2x higher returns under fixed exploration budgets compared with standard baselines, and it uses roughly 100x fewer interactions than RL baselines. The asymmetry is the point — a privileged expert provides the action labels, and a history-conditioned student learns to explore from them, so the exploration behavior is acquired through distillation rather than through raw trial and error. m:…3169

What makes the result worth a developer's attention is that it did not stay in simulation. The paper reports that policies trained in simulation transferred to real-world robot manipulation with a +40% success rate increase over standard distillation. That suggests the distilled exploration behavior is not just a simulation artifact; it survives contact with physical hardware. Instead of learning to explore through millions of rollouts, the agent learns an exploration policy from distilled guidance and then explores at test time within its context. If the numbers hold up under independent replication, exploration becomes something you can teach rather than something you have to pay for in environment steps. m:…3169

The Bayesian story doesn't hold

A second preprint pulls in the opposite direction — it subjects a popular theoretical story about in-context RL to measurement, and the story fails. The assumption under test is that Decision-Pretrained Transformers (DPT), which perform in-context reinforcement learning, behave as Bayesian posterior samplers: that they make decisions the way an optimal Bayesian agent would, updating beliefs about the task as evidence accumulates in context. The authors tested two necessary conditions for that behavior — sufficient-statistic invariance and the martingale property — and the DPT models failed both in practice. The conclusion is direct: ICRL models do not behave as Bayesian posterior samplers in practice. m:…0916

This matters because a fair amount of agent design quietly assumes something like Bayesian optimality. If you build a decision system on top of in-context RL and reason about it as though it were a posterior sampler — expecting it to explore optimally, to balance uncertainty correctly, to converge the way the theory says — you are designing against an assumption the measurements reject. The practical implication is to treat in-context RL as an empirical object, not a theoretical one: measure the decision behavior of the specific model in the specific task environment rather than inheriting guarantees from a theory that does not hold. That is a harsher, more expensive discipline, but it is the one the evidence supports. m:…0916

When in context beats fine-tuning

The third preprint tackles the adjacent question that every developer eventually faces: given a dataset, should you handle it with in-context learning or with fine-tuning? The authors propose a certified information-theoretic rule based on the information value of data, defined as the code length saved on held-out queries. The idea is that whichever approach compresses held-out queries better — the one that saves more code length — is the approach the data wants. Tested on GPT-2 query streams, the valuation method correctly identified the superior learning approach in 96.8% of cases, and a label-free valuation of support data recovered published evidence adjudicators with an AUROC of 0.91. m:…5417

The appeal here is operational. The ICL-versus-fine-tuning choice is usually made by intuition, habit, or whatever the last project did; a certified rule turns it into a measurement. If you can compute the code-length savings on held-out queries, you can automate the decision per dataset instead of per hunch, which is exactly the kind of thing a production pipeline wants. The AUROC result adds a second use: the same valuation machinery can recover evidence adjudicators without labels, which points toward data valuation as a diagnostic tool — not just which method wins, but which data is worth paying attention to. Together with the other two preprints, a pattern emerges: the in-context paradigm is growing a measurement layer. Exploration can be distilled and its cost measured; Bayesian claims can be tested and rejected; the ICL-versus-fine-tuning boundary can be certified. m:…5417

What this means for builders

Read together, the three papers sketch a maturing discipline. In-context learning is no longer just a capability you demonstrate; it is a behavior you quantify, teach, and audit. ASTEROID suggests exploration is teachable through distillation at a fraction of RL's interaction cost. The posterior-sampler study warns that the theory around in-context RL is ahead of the measurements, and that agent designers should verify rather than assume optimality. And the information-value rule gives teams a principled way to decide where fine-tuning earns its keep and where in-context learning is enough. All three are preprints, not peer-reviewed results, so the responsible move is to treat the numbers as reported findings to be replicated, not as settled facts. But the direction is consistent: the in-context paradigm is being put under the microscope, and it is surviving the scrutiny in the places that matter. m:…3169 m:…0916 m:…5417