---
title: Put the small model first — but check its work
description: In preprints, a 7-billion-parameter text-to-SQL model cuts API cost by a third when paired with a big model, while a memory method lifts accuracy and cuts tokens by three quarters. The catch: small models fail silently.
canonical: https://hangingcontext.com/stories/2026-11-17/small-models-cheap-only-if-right/
last-updated: 2026-11-17
url: https://hangingcontext.com/stories/2026-11-17/small-models-cheap-only-if-right/
published: 2026-11-17T00:58:00Z
edition: 2026-11-17T00
newsroom: artificial-intelligence
beat: research
trust: reported
source_count: 4
cited_manifest_ids: ['1794873860009625582', '1794814527845108825', '1794814527845108822', '1794851889408415228']
---

# Put the small model first — but check its work

*In preprints, a 7-billion-parameter text-to-SQL model cuts API cost by a third when paired with a big model, while a memory method lifts accuracy and cuts tokens by three quarters. The catch: small models fail silently.*

Reported · 4 Manifests · Artificial Intelligence · Published 2026-11-17 00:58 UTC

## Key facts

- In a preprint, LuthorSQL, a 7-billion-parameter text-to-SQL model, reaches 68.03% execution accuracy on BIRD's hidden test set. (self-reported) [cites: 1794873860009625582]
- In a preprint, LuthorSQL paired with Claude Opus 5.5 cuts API cost by 33.3% at the same accuracy. (self-reported) [cites: 1794873860009625582]
- In a preprint, MEM-SQL raises mean execution accuracy from 37.1% to 50.0%, with 76.6% fewer input tokens and 68.8% lower latency on average. (self-reported) [cites: 1794814527845108825]
- In a preprint, MMFlash achieves lossless decoding speedups of up to 3.58 times for vision-language models. (self-reported) [cites: 1794814527845108822]

## The gate: a small model first, the big model only when unsure

LuthorSQL is a non-autoregressive 7-billion-parameter text-to-SQL model that serves as a rapid System 1 gate for a slower LLM agent, the System 2, escalating queries only when uncertain. That is the design in a preprint from its authors, and the numbers are strong for a model this size. On its own, LuthorSQL reaches 68.03% execution accuracy on BIRD's hidden test set, and it maintains its accuracy in 9 GB with 8-bit quantized weights. It also transfers zero-shot to Snowflake, BigQuery, MySQL, PostgreSQL and DuckDB within 3.1 points of its native SQLite accuracy. [cites: 1794873860009625582]

The point of the gate is not to replace the big model but to spare it. Paired with Claude Opus 5.5, LuthorSQL cuts API cost by 33.3% at the same accuracy. The same paper carries a warning for anyone who trusts public benchmarks too much: its contamination probing shows that Opus reproduces 48 to 92% of public benchmark gold queries verbatim without seeing the schema. A gate only saves money if the test it passed was honest. [cites: 1794873860009625582]

## Memory: reuse what already worked

A second preprint attacks the same cost problem from the memory side. MEM-SQL stores whole-query templates, reusable SQL blocks and mappings from domain concepts to schema elements, and consults the schema only when no layer qualifies or the generator judges retrieved evidence insufficient. Its authors introduce HARE, a history-aware benchmark of 1,600 requests derived from a fixed pool of 823 verified queries, to measure the accuracy and cost of adapting historical SQL. On HARE, mean execution accuracy rises from 37.1% for direct generation without history to 50.0% for MEM-SQL, with 76.6% fewer input tokens and 68.8% lower latency on average. [cites: 1794814527845108825]

> **From Synorb** — Ask what Synorb covers. The MCP tool synorb-stream-search answers questions like "do you have this publisher?" without using any Manifest allowance. [Connect and ask](https://synorb.com/connect/start)


Memory has a limit, and the paper is honest about it. Further analysis shows that requests reusing several verified queries remain a bottleneck, with accuracy falling from 56.7% to 29.7%, most sharply when reused components must be placed in a new query structure or combined with a shared join. Reusing one verified piece is easy; combining several into a new shape is where the method breaks. That is a useful boundary for a developer: memory helps most on repeated, similar questions, and least on novel combinations. [cites: 1794814527845108825]

## Speed: faster decoding, and the silent-error catch

The third piece of the stack is decoding speed. Speculative decoding accelerates generation by letting a lightweight drafter propose tokens for parallel verification by a target model, but for vision-language models the drafter struggles where the image matters most: the authors of a preprint called MMFlash find acceptance rates decrease from 46% in the lowest visual-dependence quartile to 22% in the highest. Their framework adaptively upweights visually dependent tokens that the drafter predicts poorly, and for long video inputs uses sparse attention to reduce redundant computation. The result is lossless decoding speedups of up to 3.58 times, outperforming EAGLE-3 by up to 195.4% and DFlash by 17.1%. [cites: 1794814527845108822]

The catch for the whole stack comes from a fourth preprint, on small fine-tuned parsers. A small parser that emits a well-formed but wrong tree fails silently: nothing downstream can tell. The authors show that writing the target in ordinary words is the most accurate format on both models they tested, 87.5% against 85.8% exact match for the native s-expression on Qwen, and that it cuts the silent error rate by 2.3 times on Qwen and 1.8 times on LFM2. Shortening alone never helps. Strict refusal has a measured price: 2.1 to 10.1% of real, mostly polite, orders are refused, though the models learn to refuse off-menu input, handling 95.9 to 99.0% of held-out cases. The decision for a developer is the shape of the stack: small model first, memory second, fast decoding third, and a verification step that catches the errors no downstream system can see. [cites: 1794851889408415228]

## What is still unresolved

- Whether LuthorSQL's cascade savings hold on databases and question types beyond those tested.
- Whether the word-encoding fix for silent errors generalises beyond the two small models tested.

## Decision card

- **Useful for:** Cutting API cost on structured query and generation workloads; Designing a small-model-first serving stack
- **Not for:** Replacing the large model entirely — the gate depends on escalating when uncertain; Unverified outputs in high-stakes settings
- **Compare against:** Direct generation with a frontier model on every query; Standard speculative decoding baselines like EAGLE-3

## Provenance ledger

| Manifest id | Title | Source | Claims used | Published |
|---|---|---|---|---|
| `1794873860009625582` | LuthorSQL: A One-Pass Text-to-SQL Model as System 1 for a Slower LLM Agent | [huggingface-daily-papers](https://huggingface.co/papers/2611.17780) | 1794874032321629428, 1794874032325629439, 1794874032319629418, 1794874032323629432, 1794874032324629435, 1794874032327629444 | 11-16 00:00 |
| `1794814527845108825` | MEM-SQL: Memory-Empowered Text-to-SQL | [huggingface-daily-papers](https://huggingface.co/papers/2611.18338) | 1794815181972126441, 1794815181974126442, 1794815181975126443, 1794815181968126438, 1794815181971126440, 1794815181965126436, 1794815181966126437, 1794815181977126444 | 11-16 00:00 |
| `1794814527845108822` | MMFlash: Faster Speculative Decoding for Vision-Language Models | [huggingface-daily-papers](https://huggingface.co/papers/2611.18482) | 1794815185647126648, 1794815185635126640, 1794815185641126644, 1794815185644126646, 1794815185645126647, 1794815185648126649 | 11-16 00:00 |
| `1794851889408415228` | Parsed, Plausible, Wrong: Output Format Controls Silent Errors in Small-Model Semantic Parsing | [huggingface-daily-papers](https://huggingface.co/papers/2611.17169) | 1794852157296423380, 1794852157323423394, 1794852157326423402, 1794852157316423391, 1794852157330423411, 1794852157328423408 | 11-16 00:00 |

## Run these Manifests yourself

Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.

1. **Get a Synorb Key.** Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{"client":"your-agent-name"}`. The key is the `api_key` field of the answer. Save the key. Existing 72 Hour Agent Demo keys keep their issued limits. New keys require email OTP verification. An existing demo key lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.
2. **Read each cited Manifest by id.** REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief.
3. **Check the claim.** Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them.

Cited Manifest ids:

- `1794873860009625582`
- `1794814527845108825`
- `1794814527845108822`
- `1794851889408415228`

```sh
for id in 1794873860009625582 1794814527845108825 1794814527845108822 1794851889408415228; do
  curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id"
done
```

> **From Synorb** — Run the cited Manifests yourself. Every claim in this paper points to a Manifest id. Get a Synorb Key and read the Manifests behind it. Existing 72 Hour Agent Demo keys keep their issued limits. New keys require email OTP verification. An existing demo key lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons. [Get a key](https://synorb.com/connect/start)

## Related stories

- [Three inference providers disclosed prompt failures in the same week](https://hangingcontext.com/stories/2026-11-22/where-your-prompts-actually-go/)
- [Three preprints fix the plumbing: which data to keep, which causes matter, when robots must stop](https://hangingcontext.com/stories/2026-11-22/the-weeks-plumbing-papers/)
- [The agent platform is standardising: protocols, observability, governance](https://hangingcontext.com/stories/2026-11-22/the-agent-platform-standardizes/)
- [Four patches, one lesson: the AI stack's weak points are the boring parts](https://hangingcontext.com/stories/2026-11-22/patch-the-ai-stack/)

## Links

- HTML: https://hangingcontext.com/stories/2026-11-17/small-models-cheap-only-if-right/
- JSON: https://hangingcontext.com/stories/2026-11-17/small-models-cheap-only-if-right/index.json
- Markdown: https://hangingcontext.com/stories/2026-11-17/small-models-cheap-only-if-right/index.md

Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.

---
Reuse (terms v1): Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product. Credit: "Hanging Context, with provenance by Synorb", with a link to https://hangingcontext.com/stories/2026-11-17/small-models-cheap-only-if-right/. Terms: https://hangingcontext.com/terms/reuse/
