The self-improvement pitch is running ahead of its guardrails

The demos are real this time

In a preprint, the Athanor authors describe an agent-native system that hands frontier models full research autonomy over post-training: data selection, algorithm modification, hyperparameter tuning, and weight merging, all inside a sandbox with low-friction engineering tools. In their reported runs, Claude Opus 5.5 agents took Qwen3-4B-Base from 15.9 to 67.3 on a six-task development suite, and agent-led work spanning supervised fine-tuning, reinforcement learning, weight merges, and self-distillation produced models matching or surpassing the official instruct-tuned versions. That is the whole post-training job, executed by agents rather than researchers. m:…9020

NeoHorse-1 pushes the same idea one level deeper. In a second preprint, its authors close an evaluation-selection-update loop: interaction logs become structured training data, a routing harness allocates capability demand across a heterogeneous model pool, and routing-guided on-policy distillation trains the student. The reported result is a 5.93-point macro-average benchmark gain for 4B models, narrowing the gap to larger models, alongside a prefill-only NeoHorse-Jev-4B architecture built for structured decision-making in agent workflows. The pitch writes itself: models that improve themselves, cheaper than hiring the researchers. m:…8772

The theory says full autonomy cannot be verified

The counterweight is a theory preprint. In a theory preprint, the authors model agents as programs and show that no terminating procedure can simultaneously guarantee reliability, meaning never accepting a non-improvement, and non-conservatism, meaning never missing an available improvement. Perfect automated verification of semantic improvement is impossible in the general case. The automated gatekeeper you would need to trust an unbounded self-improvement loop cannot exist. m:…9926

There is a decidable regime: bounded evaluation and finite proposal budgets, where exhaustive simulation yields a terminating search procedure. That is the practical world of scorecards, budgets, and fixed evaluation sets. But the paper's sting is that a chain of accepted updates can raise the bounded score in every round while the true semantic score falls. Bounded evaluation makes the search decidable; it does not make the scoreboard honest. m:…9926

Deployment adds a failure mode with data behind it

A third preprint, on skill-policy co-evolution, documents what happens after the demo ends. When agents co-evolve a policy with a specific skill bank, the policy's competence stays tied to that bank, creating fragility to skill drift. Under drift, worst-case success rates land 5 to 13 points below peak performance, and the drift erases 52 to 93 percent of the performance gain over no-skill baselines. Standard evaluation protocols do not capture this: the agent looks fine in the lab and degrades in production as its toolset evolves independently of the policy model. m:…5536

Put together, the cautionary results form a coherent risk model: the verifier cannot be perfect in general, the bounded score can rise while real quality falls, and the skills the agent trained with will drift. None of this says self-improvement fails. It says the loop needs instrumentation designed for adversaries, not averages. m:…9926 m:…5536

What a careful operator builds first

The decision that follows is straightforward, and this is reasoning rather than a reported result: keep evaluation bounded, keep proposal budgets finite, and gate every update on re-verification against fresh skills rather than the bank the policy trained with. Treat the bounded score as a hypothesis to be re-tested, not a verdict. And ask vendors of agent platforms for their skill-drift audits before believing their improvement curves. m:…9926 m:…5536

The second-order point is about who compounds the advantage. The teams that master bounded, audited self-improvement get models that keep getting cheaper and better. The teams that trust the loop blindly get score drift with production incidents attached. The demos show the capability is real; the theory shows the guardrails are the actual product. m:…9020 m:…9926