Put the small model first — but check its work
In preprints, a 7-billion-parameter text-to-SQL model cuts API cost by a third when paired with a big model, while a memory method lifts accuracy and cuts tokens by three quarters. The catch: small models fail silently.
- In a preprint, LuthorSQL, a 7-billion-parameter text-to-SQL model, reaches 68.03% execution accuracy on BIRD's hidden test set. Self-reported m:…5582
- In a preprint, LuthorSQL paired with Claude Opus 5.5 cuts API cost by 33.3% at the same accuracy. Self-reported m:…5582
- In a preprint, MEM-SQL raises mean execution accuracy from 37.1% to 50.0%, with 76.6% fewer input tokens and 68.8% lower latency on average. Self-reported m:…8825
- In a preprint, MMFlash achieves lossless decoding speedups of up to 3.58 times for vision-language models. Self-reported m:…8822
The gate: a small model first, the big model only when unsure
LuthorSQL is a non-autoregressive 7-billion-parameter text-to-SQL model that serves as a rapid System 1 gate for a slower LLM agent, the System 2, escalating queries only when uncertain. That is the design in a preprint from its authors, and the numbers are strong for a model this size. On its own, LuthorSQL reaches 68.03% execution accuracy on BIRD's hidden test set, and it maintains its accuracy in 9 GB with 8-bit quantized weights. It also transfers zero-shot to Snowflake, BigQuery, MySQL, PostgreSQL and DuckDB within 3.1 points of its native SQLite accuracy. m:…5582
The point of the gate is not to replace the big model but to spare it. Paired with Claude Opus 5.5, LuthorSQL cuts API cost by 33.3% at the same accuracy. The same paper carries a warning for anyone who trusts public benchmarks too much: its contamination probing shows that Opus reproduces 48 to 92% of public benchmark gold queries verbatim without seeing the schema. A gate only saves money if the test it passed was honest. m:…5582
Memory: reuse what already worked
A second preprint attacks the same cost problem from the memory side. MEM-SQL stores whole-query templates, reusable SQL blocks and mappings from domain concepts to schema elements, and consults the schema only when no layer qualifies or the generator judges retrieved evidence insufficient. Its authors introduce HARE, a history-aware benchmark of 1,600 requests derived from a fixed pool of 823 verified queries, to measure the accuracy and cost of adapting historical SQL. On HARE, mean execution accuracy rises from 37.1% for direct generation without history to 50.0% for MEM-SQL, with 76.6% fewer input tokens and 68.8% lower latency on average. m:…8825
Memory has a limit, and the paper is honest about it. Further analysis shows that requests reusing several verified queries remain a bottleneck, with accuracy falling from 56.7% to 29.7%, most sharply when reused components must be placed in a new query structure or combined with a shared join. Reusing one verified piece is easy; combining several into a new shape is where the method breaks. That is a useful boundary for a developer: memory helps most on repeated, similar questions, and least on novel combinations. m:…8825
Speed: faster decoding, and the silent-error catch
The third piece of the stack is decoding speed. Speculative decoding accelerates generation by letting a lightweight drafter propose tokens for parallel verification by a target model, but for vision-language models the drafter struggles where the image matters most: the authors of a preprint called MMFlash find acceptance rates decrease from 46% in the lowest visual-dependence quartile to 22% in the highest. Their framework adaptively upweights visually dependent tokens that the drafter predicts poorly, and for long video inputs uses sparse attention to reduce redundant computation. The result is lossless decoding speedups of up to 3.58 times, outperforming EAGLE-3 by up to 195.4% and DFlash by 17.1%. m:…8822
The catch for the whole stack comes from a fourth preprint, on small fine-tuned parsers. A small parser that emits a well-formed but wrong tree fails silently: nothing downstream can tell. The authors show that writing the target in ordinary words is the most accurate format on both models they tested, 87.5% against 85.8% exact match for the native s-expression on Qwen, and that it cuts the silent error rate by 2.3 times on Qwen and 1.8 times on LFM2. Shortening alone never helps. Strict refusal has a measured price: 2.1 to 10.1% of real, mostly polite, orders are refused, though the models learn to refuse off-menu input, handling 95.9 to 99.0% of held-out cases. The decision for a developer is the shape of the stack: small model first, memory second, fast decoding third, and a verification step that catches the errors no downstream system can see. m:…5228
What is still unresolved
- Whether LuthorSQL's cascade savings hold on databases and question types beyond those tested.
- Whether the word-encoding fix for silent errors generalises beyond the two small models tested.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.