Scaling laws get sharper: MoE capacity theory, VLM transfer prediction, and a 20k-parameter jet tagger

MoE: two dials, not one

One preprint gives mixture-of-experts transformers something they have mostly lacked: a formal generalization theory. The core move is to derive MoE generalization bounds by separating active per-input capacity from routing combinatorics. That separation matters because it untangles two different costs: the compute spent activating experts for a given input, and the combinatorial overhead of deciding where to route it. Developers tuning MoE architectures have long conflated the two; this work treats them as independent dials. m:…5401

With active parameters accounted for, the paper finds that approximation and estimation in MoE networks trade off much like they do in dense networks. Error can fall either by scaling active capacity or by adding experts, and which one works depends on the dominant bottleneck. The practical read, offered here as reasoning rather than reported fact: before adding experts, check whether routing or capacity is what actually limits you. That is a capacity-planning question, not a parameter-counting one. For infrastructure teams pricing MoE deployments, the distinction changes the cost model: capacity is the part you provision, routing is the part you engineer. m:…5401

VLM: predict vision scores from text

A second preprint tackles a different scaling question: what transfers from text to vision? The authors propose a capability-driven multimodal scaling law that predicts vision-language model benchmark accuracy directly from observable textual capability scores. If the law holds, developers could estimate multimodal performance before training, cutting the need for expensive empirical sweeps when choosing a backbone. The promise is cheaper model selection: score the text backbone you already have, and get a read on the vision-language model you have not trained yet. m:…5630

Two findings from that paper deserve attention. First, base LLMs generally outperform instruction-tuned counterparts as VLM backbones, which the authors attribute to higher absorption rates and lower data-scaling decay. That is an argument for starting from the base model, not the chat model, when building multimodal systems. Second, certain textual benchmarks negatively correlate with multimodal performance, which the authors read as evidence of latent benchmark-gaming: climbing those leaderboards can actively hurt vision performance. The implication is uncomfortable but useful: not every text capability transfers, and some transfer backwards. m:…5630

Distillation: tiny students, strong teachers

The third preprint asks how far knowledge distillation can be pushed for jet tagging, the particle-physics task of classifying collision jets. The authors report that distillation consistently improves student performance regardless of architecture, with the gains scaling with teacher strength. The bottleneck is the teacher, not the student's shape. m:…2021

The headline numbers: a distilled 20,000-parameter L-GATr-slim model outperforms every published jet tagger trained from scratch, and a 2,000-parameter distilled model matches the ParT model. The paper also notes that inference speed can vary by nearly an order of magnitude between models of the same parameter size, so parameter count alone is a poor guide to deployment cost. For teams deploying AI in resource-constrained settings like real-time physics experiments, the lesson is to distill from the strongest teacher available rather than train bigger from scratch. Put differently, a strong teacher plus a small student can beat a big model trained from scratch: an efficient route to capability when inference budgets are tight. m:…2021