Decision models get measured: a launch, a calibration benchmark, and a robustness audit

The product: d1-ultra

Liquid AI released d1-ultra, a new multimodal decision model built, the company says, for harder, messier real-world use cases like defect detection. The company says most decision models are overconfident, so it specifically trained d1-ultra to produce more reliable probabilities, and based the model on a larger open-weight model with its own customizations. m:…8791

On performance, Liquid AI says d1-ultra outperforms Jev 1.14 across all tasks in 112 evaluations and leads every other decision model on the Artificial Analysis Decision Index by at least 8 points. The pricing is designed for high-volume classification: $0.042 per million input tokens on the Liquid API, billed on input tokens only with no output tokens, images counted at 1.5 tokens per 32x32-pixel patch. The company says the model will soon be available through Vercel and OpenRouter. m:…8791

The measurement: calibration

In a preprint posted this week, researchers introduced ProbabilityBench, a benchmark that supplies exact, instance-level class probabilities across three simulated text domains. By defining the full data-generating process for each environment, the authors compute both exact class probabilities and Bayes error analytically, which lets them measure reported probability quality directly with KL divergence rather than guessing at it. m:…8802

The study evaluated 16 models and found that decision models frequently sit on the observed cost-quality Pareto frontiers, achieving strong trade-offs between probability accuracy and cost. The benchmark also works as a meta-evaluation tool, using its known ground truth to check the accuracy of standard calibration metrics and Bayes-error estimators themselves. This is the missing yardstick the category needed: if you buy a decision model for its probabilities, this is how you check whether the probabilities mean anything. m:…8802

The audit: robustness

A second preprint, DecisionRobust, takes the other side of the question: what happens when someone tries to break the decision model. The benchmark evaluates JEV-style decision models against adversarial text, image, and prompt injection attacks, with a public release of 1,058 curated attack examples across five suites, testing 30 model configurations on text tasks and 17 on vision tasks. m:…5027

Two findings stand out. First, resistance to input perturbations does not necessarily imply resistance to prompt injection, which means the usual robustness testing can miss the attack vector decision models are most likely to face in production. Second, label-consistency checks, which verify that an attack preserves the correct answer, substantially reduce measured attack success, and attacks expose vulnerabilities beyond the models used to construct them. The implication is that teams deploying decision models should test them like agents, not like classifiers. m:…5027

The frontier: AutoThink

A third preprint, StartLux-Decision, pushes the category itself forward with AutoThink models from 0.8B to 27B parameters that invoke explicit reasoning only for uncertain decisions and answer confident cases directly. The 27B AutoThink model reached 73.4 on the public subset of Decision Index 0.3, while the Instant single-pass variants beat comparable open-weight decision models by 7.4 to 16.9 points. m:…0398

The training method, RLCD-squared, uses a frozen model and proper scoring rules to reward better decision probabilities and applies a soft penalty to overlong reasoning traces, which cut average reasoning length by up to 65% while largely preserving accuracy and raised the share of traces terminating within the generation limit by up to 48 percentage points. The models run on vLLM, llama.cpp, and MLX, with quantized variants for lower memory. Taken together, this suggests decision models are becoming a real product category: priced per decision, benchmarked on calibration, audited for robustness, and optimized to think only when it pays. m:…0398