Models that learn on the job: the pitch is specialisation, and the research is catching up
Trajectory's Ronak Malde argues continual learning is the next paradigm, citing a Harvey model six times cheaper than frontier models, while preprints probe sparsity in continual learning and the side effects of mass-produced training data.
- Trajectory's platform enabled Harvey to train a model six times cheaper than frontier models, and Harvey increased model accuracy from 4% to 18% using the platform, according to Trajectory co-founder Ronak Malde. Self-reported m:…7593
- Only 1% of enterprises have adopted AI, largely due to the difficulty of automating manual workflows, according to Malde. Self-reported m:…7593
- In a preprint, weighted L2 regularisation preserves sparsity bias in continual learning for diagonal linear networks, and converges faster than standard L2 or L1 for sufficiently sparse teachers. Self-reported m:…4205
- In a preprint, frontier models outperform humans at predicting unintended training side effects, and datasets with surprising side effects generalise across model families and sizes. Self-reported m:…6009
Pseudo-continual is the status quo
Ronak Malde, co-founder of Trajectory, argues that the next paradigm in AI is continual learning, where models improve based on real-world production data. He says current AI development is in a state of pseudo-continual learning, where updates require mass batch processing rather than real-time iteration. Trajectory aims to bridge the gap between foundation models and domain-specific needs by simplifying training, evaluation, and deployment, focusing on trajectories as the unit of work. The future, in Malde's account, is a shift toward specialisation, where companies customise their own models and harnesses rather than renting a frontier model for everything. m:…7593
The evidence Malde offers is specific. Trajectory's platform enabled Harvey to train a model that is six times cheaper than frontier models, and Harvey increased model accuracy from 4% to 18% using the platform, according to Malde. Rogo is using Trajectory to train a finance agent that is more efficient and concise than frontier models. The numbers are the vendor's own account, and the 4% to 18% jump is a large one from a low base, but the claim is testable: a specialised model, trained on the company's own trajectories, beating the general model on the company's own task at a sixth of the cost. m:…7593
Malde also gives a reason the shift has been slow. Only 1% of enterprises have adopted AI, largely due to the difficulty of automating manual workflows, according to him. The bottleneck is not the model. It is the workflow. That is the same argument Basis made this week about process, from a different company: the hard part is making the work repeatable enough to learn from. m:…7593
The research underneath
In a preprint posted this week, researchers address a specific failure in continual learning: the degradation of sparsity bias when using standard L2 regularisation. By proposing a weighted L2 variant for diagonal linear networks, the authors enable the preservation of sparse feature learning in sequential tasks. Weighted L2 regularisation preserves sparsity bias in continual learning for diagonal linear networks, according to the preprint, and the proposed variant converges faster than standard L2 or L1 regularisation for sufficiently sparse teachers. Sparse representations matter because they are cheaper to store and easier to update, which is exactly what a continually learning system needs. m:…4205
Trajectory is researching group size one algorithms to eliminate the need for massive environment replays in training, according to Malde, and uses frontier coding agents to implement system design and training infrastructure. The research direction and the product direction agree: continual learning has to get cheaper to run, or it stays a lab idea. m:…7593
Learning on the job has side effects
A second preprint is the caution. The authors present a scalable method for identifying training data that triggers unintended model behaviours, such as persona shifts or misalignment, by automating the discovery of datasets with surprising side effects. Automated discovery of these datasets is feasible using LLM judges, evolution, reinforcement learning, and subliminal learning, according to the preprint. Frontier models outperform humans at predicting unintended training side effects, and the datasets generalise across different model families and sizes. m:…6009
The implication for continual learning is direct. A model that learns from production data will also learn the side effects in that data, and the preprint suggests those side effects transfer across models. Malde argues continual learning is a safer approach to AI improvement than recursive self-improvement because it keeps humans in the loop, and notes that language models currently lack a way to encode transition functions, which is a barrier to autonomous agent curiosity. The safety argument is that humans stay in the loop. The research argument is that the loop needs to watch for side effects, because the model will not announce them. m:…7593 m:…6009
What is still unresolved
- Whether Harvey's 4% to 18% accuracy gain and 6x cost advantage hold on other customers' tasks
- Whether weighted L2 sparsity results transfer from diagonal linear networks to production models
- How to detect surprising side effects in live production data, not just in curated datasets
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.