Kubernetes requests drift upward, and AI workloads pay the bill
CPU overprovisioning rose from 40% to 69% in a year. Polyaxon shows how agent workers and sandboxes need their own budgets for scaling and scratch disk.
- The 2026 State of Kubernetes Optimization Report indicates CPU overprovisioning rose from 40% to 69% year over year. Self-reported m:…7366
- KEDA 2.21 can scale SQS-backed worker pools based on both visible and in-flight messages. Self-reported m:…6988
- Temporary files can cause pod eviction if they exceed disk budgets, and Kubernetes uses emptyDir.sizeLimit to control scratch volume usage. Self-reported m:…6986
The drift
Kubernetes scheduling runs on requests: a pod asks for a certain amount of CPU and memory, and the scheduler places it where that capacity exists. The problem, described by Cast AI, is that those requests drift upward over time. Request drift is driven by defensive estimates, copied manifests, and incident-driven increases that are not revisited, so each incident leaves the request a little higher and nobody brings it back down. Kubernetes request overprovisioning occurs when pod CPU or memory requests exceed the capacity a workload needs, and the 2026 State of Kubernetes Optimization Report indicates CPU overprovisioning rose from 40% to 69% year over year. m:…7366
The cost is not just the extra capacity on paper. Inflated requests reduce pod density on nodes, forcing the autoscaler to provision unnecessary capacity, so a cluster that looks full is mostly holding reservations for work that never runs. The fix the company proposes is measurement: provisioned, requested, and used capacity should be measured separately, because each answers a different question about where money goes. A request-to-P95 CPU ratio above 2x is a signal for review, which gives teams a concrete threshold instead of a vague feeling that requests look high. m:…7366
A common fear about cutting requests turns out to be misplaced. Exceeding a memory request does not trigger an OOM kill by itself, so trimming an inflated request does not automatically make a workload fragile, though the company notes that lowering a CPU request can increase reported utilization for HPA, potentially triggering additional replicas. The point is that requests are a scheduling input, not a safety guarantee, and treating them that way leads teams to over-provision for protection they never receive. m:…7366
Scaling agent workers safely
AI agent workers add a wrinkle that ordinary web services do not have: a worker that is killed mid-task can lose the task. Polyaxon describes autoscaling AI agent workers on Kubernetes with KEDA, and notes that KEDA 2.21 can scale SQS-backed worker pools based on both visible and in-flight messages. Standard SQS scalers that only look at available messages may prematurely scale down busy workers, because a message that has been claimed by a worker is invisible to them even though the work is still running. m:…6988
The guidance is practical rather than theoretical. Setting scaleOnInFlight to true in KEDA allows the scaler to include claimed messages in the target count, and workers should be designed to stop polling or cancel outstanding receives upon receiving a termination signal, so a scale-down does not strand work. TerminationGracePeriodSeconds should be set to cover the time required to stop polling and checkpoint tasks, and the SQS ApproximateAgeOfOldestMessage metric is useful for detecting growing backlogs. Hard node failures can skip graceful shutdown, necessitating durable writes before acknowledging task completion, and a requests-per-minute limit requires rate control within the application, even with autoscaling. m:…6988
Sandboxes need disk budgets
Another resource trap is disk, and it hits coding agents hardest. Coding agents often download archives and extract repositories, creating temporary files that consume disk space, and temporary files can cause pod eviction if they exceed disk budgets. Kubernetes uses emptyDir.sizeLimit to control scratch volume usage, and it uses ephemeral-storage requests to schedule pods on nodes with sufficient capacity, while ephemeral-storage limits trigger pod eviction when the limit is passed. m:…6986
Polyaxon recommends an additional 1 GiB for logs and files outside the scratch path, and notes that pod eviction while scratch is below 4 GiB suggests usage is occurring in writable layers or logs rather than in the scratch volume itself. EmptyDir volumes survive container restarts within the same pod, and Kubernetes removes emptyDir volumes when the pod is removed from the node, so scratch is temporary by design and teams should write outputs to a configured path and confirm artifact storage. The broader lesson across these traps is the same: agent workloads fail on resources that web workloads rarely stress, and the fix is to budget each explicitly instead of discovering the limit by eviction. m:…6986
What is still unresolved
- How much of the drift comes from AI agent workloads specifically
- Whether autoscalers tuned for agents differ from web workloads
- How teams audit requests without breaking running agents
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.