Sub-2B models are doing real work: permissible-data reasoning, Vietnamese and Arabic distillation, edge compression
A sub-2B model trained on permissible data matches 9B models, distillation compresses Vietnamese and Arabic models nearly losslessly, and distillation wins again on edge hardware.
- In a preprint on Mimir v1.5, a sub-2B hierarchical reasoning model trained on permissible data, the model tops all sub-2B models across 20 benchmarks in English, Danish, and Math & Code, and matches or exceeds 9B models (Qwen 3.5 9B, Gemma 4 E4B) on English and Danish averages. Repeated training on 161 curated, permissible datasets does not raise memorization risk. Self-reported m:…2613
- In a preprint introducing GoiBERT, compact Vietnamese models built via two-stage knowledge distillation, GoiBERT-large leads non-teacher models across eight Vietnamese benchmarks; GoiBERT-base retains 99% of the strongest compact baseline's performance at 25% of its size; and the models run directly on raw text, eliminating external word segmentation. Self-reported m:…6735
- In a preprint introducing AQL-Lite, a data-centric continual pretraining recipe for Arabic LLMs, the method achieves 94% of full-parameter CPT performance in 52% of the wall-clock time. CPT on AQL-curated data raises a four-benchmark Arabic mean by 3.51 points over CPT on raw web text, which the authors report yields no gain over the base model at all. Self-reported m:…9269
- In a preprint benchmarking model compression across heterogeneous edge platforms for a predictive maintenance use case, knowledge distillation was the most effective compression technique for CNNs: distilled models achieved over 90% speedup across nine heterogeneous edge platforms, with a 98.9% size reduction while maintaining a 91.22% balanced F-score. Self-reported m:…4610
Reasoning under two billion
The most striking paper of the week is Mimir v1.5, a sub-2B hierarchical reasoning model trained on permissible data, presented in a preprint this week. The authors report that it outperforms all sub-2B models across 20 benchmarks in English, Danish, and Math & Code, and that it matches or exceeds the performance of 9B-parameter models, specifically Qwen 3.5 9B and Gemma 4 E4B, on English and Danish averages. A model a fifth the size is grading like its big sibling. m:…2613
The training recipe matters as much as the score. Multi-epoch training on a curated, permissible mixture of 161 datasets does not increase memorization risk, according to the authors. For founders worried about the legal exposure of web-scale training data, the claim is that careful curation plus repetition gets you 9B-class reasoning without the provenance headache. m:…2613
Languages that aren't English
Vietnamese is the week's second data point. A preprint introduces GoiBERT, compact Vietnamese language models built through two-stage knowledge distillation. GoiBERT-large achieves the highest average score among non-teacher models across eight Vietnamese benchmarks, while GoiBERT-base retains 99% of the performance of the strongest compact baseline at 25% of its size. Compression here is nearly a free lunch. m:…6735
There is a practical detail that matters more than the benchmarks: GoiBERT models operate directly on raw text, eliminating the need for external word segmentation. Vietnamese tokenization has long been a deployment tax; removing it means one less dependency, one less failure mode, one less thing to maintain. m:…6735
Arabic gets the same treatment with a different recipe. AQL-Lite, a data-centric continual pretraining method presented in a preprint this week, achieves 94% of full-parameter CPT performance in 52% of the wall-clock time. The lever is data quality: CPT on AQL-curated data raises a four-benchmark Arabic mean by 3.51 points over CPT on raw web text. m:…9269
The negative result is the one to remember: raw web text CPT yields no gain over the base model for Arabic adaptation, the authors report. The tokens are not the work; the curation is. For anyone budgeting a language adaptation run, the preprint says to spend on the data pipeline before the GPU hours. m:…9269
The edge check
The fourth preprint closes the loop on deployment. Benchmarking model compression techniques across heterogeneous edge platforms for a predictive maintenance use case on rotating machinery, the authors find knowledge distillation is the most effective compression technique for CNNs. The distilled models achieved over 90% speedup across nine heterogeneous edge platforms, with a 98.9% size reduction while maintaining a 91.22% balanced F-score. Distillation keeps winning because it transfers behavior rather than compressing weights blindly, and the paper's cross-platform results suggest the win is durable across hardware generations. m:…4610
Put the week together and the pattern is a playbook. Reasoning capability now comes in sub-2B packages trained on permissible data. Languages get served by distilled specialists at a quarter the size. Edge hardware gets models compressed by two orders of magnitude with accuracy intact. The founder's question is shifting from which big model to rent to which small model to own. The economics follow: own a distilled specialist where the workload is known, rent the large generalist where it is not. This week the benchmark tables keep making that split explicit. m:…2613 m:…6735 m:…9269 m:…4610
What is still unresolved
- Mimir's 9B-class claims are the authors' own benchmark results; independent replication is not reported.
- GoiBERT's Vietnamese benchmark results cover eight benchmarks; coverage of conversational and long-context tasks is unclear.
- The edge compression study is a predictive maintenance use case; transfer to other task families is not established.
- AQL-Lite's data-curation recipe is reported at 94% of full CPT performance; the remaining gap's composition is not detailed.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.