Inference optimization moves from chips to data and self-tuning

The ground keeps moving

Together AI's case for Polymorphic Inference starts with a simple observation: 80% of OpenRouter traffic goes to models released in the last three months. When the model fleet turns over that fast, a hand-tuned engine configuration is out of date almost as soon as it ships. Polymorphic Inference is Together's name for systems that adapt to model release velocity, architecture diversity, and hardware changes instead of assuming a fixed setup. m:…2815

The old definition of inference, in Together's telling, was closed models, traditional Transformer architectures, and single-user, single-hardware configurations. Its compiler already runs in production on Together's serverless endpoints, with a limited beta for dedicated endpoints, and uses continuous compilation to adapt automatically to new traffic shapes and workloads. The company also says it served more tokens than closed-source model labs on OpenRouter in October 2026, a claim about volume that, if accurate, shows how much traffic now flows through open models. m:…2815

Databricks reports the same shift from the enterprise side. Open-weight models now account for about 40% of tokens used on its Unity Gateway, and the average customer there went from 4.2 unique models in January 2026 to 7.1. The company says open-weight models now benchmark at levels comparable to top frontier models at a lower cost, and that its gateway provides smart routing to shift routine tasks to cheaper models, with fallback chains for rate limits or outages. m:…6685

The bottleneck is data, not arithmetic

VAST Data argues the optimization target itself has moved. Traditional AI optimization focused on accelerating computation, but inference now splits into distinct prefill and decode phases with different computational characteristics, and agentic workloads add persistent context, retrieval, and multiple models running at once. In that world, the company says, optimization is shifting from compute-centric acceleration to data-centric infrastructure. m:…1877

The physical reason is blunt: increasing memory bandwidth is harder than increasing compute capacity. KV caching, which eliminates redundant prefill computation by reusing already-processed context, is the kind of data-side win VAST points to, and its engineers test configurations using LMCache, SGLang, and NVIDIA Dynamo. Infrastructure must support fleets of agents and multiple models concurrently, the company says, which means the platform holding the data matters as much as the chips doing the math. m:…1877

Databricks adds a wrinkle buyers often miss: the choice of harness, meaning prompts, tools, and control flow, significantly impacts measured model performance. A model that looks weak in one harness can look strong in another, and frontier model guardrails can block legitimate work. The company cites Hugging Face responders using GLM 5.2 to analyze attack logs after frontier models refused the task. Picking a model without fixing the harness is measuring the wrong thing. m:…6685

Capacity only pays when it is used

Aethir's warning is about the economics underneath all of this. Depreciation for GPU hardware begins the day the hardware arrives at the facility, and a fully costed cluster breaks even between 55% and 65% utilization on three-year depreciation. Monthly depreciation for an 8x H100 node runs roughly $7,000 to $10,000. The gap between go-live and steady, paying utilization is where money is lost, because depreciation outpaces revenue while the cluster ramps. m:…8504

The contrast is stark: enterprise Kubernetes clusters average 5% GPU utilization, while Aethir says its own GPU network holds 95% or higher, and the Meta Research SuperCluster reports 83% to 85%. Aethir's own rule is to build data centers only when compute demand is already secured, and it notes that NVIDIA requires a 24-hour soak test under representative inference load before a provider serves production traffic. m:…8504

Global AI capital spending is projected at $400 billion to $450 billion in 2026 and rising toward $1 trillion by 2028, according to Aethir. That scale makes the utilization question unavoidable: at these spending levels, the difference between a cluster at 5% and a cluster at 65% is not an efficiency footnote, it is the business. m:…8504