Incident Library

IL-0413 Microsoft · 2025-10-17 (published)ai-infrakubernetes

LLM inference engines timed out or ran out of resources under load in about 60% of 156 severe incidents at Microsoft

Three Microsoft engineers studied the 156 high-severity incidents (129 at severity 2 and 27 at severity 2.5, on an internal scale where 0 is most critical) of Microsoft's cloud LLM inference service from April to June 2025, using the incident management system, postmortem reports, runtime logs and dashboards, and Designated Responsible Individual (DRI) playbooks and tools. Each incident was labelled by hand with one of four root cause categories, and the labelling agreement reached a Cohen's kappa of 0.89. In the service, an API gateway receives REST or gRPC requests, an endpoint load balancer applies health probes, GPU capacity-aware routing, request batching and token caps, and inference engines on GPU clusters managed by Kubernetes load model weights, run requests and manage the KV cache and sampling parameters.

Inference engine failures were the largest category, about 60% of incidents, and were mostly driven by latency and GPU capacity: timeouts on long-running requests and resource exhaustion under load. Capacity increases, traffic routing and node restarts were applied mostly to these failures. For Case A the team enabled dynamic pushback at the load balancer with caps on pending tokens and requests, tuned request batching and GPU/CPU replica counts, improved gRPC backoff and error mapping, and limited sampling parameters with per-client throttling; availability stabilised, HTTP 429s fell and timeout-driven HTTP 5xx decreased under stress. Against timeouts on long-running requests the team used connection liveness strategies such as streaming, request batching and pending token caps, together with GPU capacity-aware routing, which correlated with fewer HTTP 408s and better SLA; for one customer the normalised HTTP 408 rate fell from 2.72% to 0.47% between February and March 2025, before the study window, which the paper reads as a directional association with the connection liveness mitigation. Across all 156 incidents ≈74% were auto-detected and ≈28% needed a hotfix, and hotfix-driven periods lengthened time to mitigate: P75 time to mitigate was ≈32 hours in April and ≈49 hours in May.

Severity
–
Scope
94 of 156 high-severity incidents (April to June 2025) of Microsoft's cloud LLM inference service were inference engine failures
Users affected
Customers of the LLM inference service; ≈96% of all incidents affected a single endpoint
Region
–
Time to detect
–
Time to resolve
–

Symptoms

In Microsoft's cloud LLM inference service, the inference engines themselves failed in 94 of the 156 high-severity incidents from April to June 2025 (≈60%). Requests timed out or ran into resource exhaustion under load, and callers got HTTP 408, 429 and 5xx errors.

Within the engine failures the paper's breakdown is timeouts ≈40%, resource exhaustion ≈29%, crashes ≈12%, latency spikes ≈6%, input fetch ≈6%, KV cache ≈5%, constraint sampling ≈1% and memory leaks ≈1%. Across all 156 incidents the error code was HTTP 500 in 115 (≈74%), HTTP 408 in 16 (≈10%), HTTP 503 in 14 (≈9%), HTTP 429 and HTTP 424 in 5 each (≈3%) and HTTP 504 in 1 (≈1%). The paper reads the dominance of HTTP 500 as monitoring gaps that let requests be routed to engines that could not serve them.

In the paper's Case A, bursty load saturated the image encoders and model containers of vision and reasoning models, with GPU/CPU imbalance and KV cache pressure, and traffic with unusually high sampling parameters overwhelmed serving, which gave HTTP 429/5xx and timeouts. In the timeout pattern, long-running requests hit frontend HTTP 408 timeouts even though the backends eventually returned HTTP 200.

Over the study window token generation grew by ≈1.27× from April to June, P75 time to first token (TTFT) rose by ≈60 ms and P75 time to last token (TTLT) by ≈80 ms, which the paper connects to emerging engine saturation, KV cache inefficiencies or architectural bottlenecks under the higher load.

Symptom Intermittent timeoutsHigh error rateHigh latency

Root cause

Root cause Capacity exhaustion

Component Model inference server
Contributing factor Monitoring blind spot
Trigger
Bursty load, traffic with unusually high sampling parameters and long-running requests pushed the inference engines into resource exhaustion and timeouts.

Detection

Across the 156 incidents, 115 (≈74%) were auto-detected, meaning health probes or DRI pipelines alerted before customers reported the problem; the others were not.

Remediation

Action items

Lessons

Hotfixes restore correctness but lengthen time to mitigate, while traffic routing and node rebalancing mitigate faster under overload and timeout patterns. Even with AIOps tooling a meaningful share of incidents still needs manual detection or judgement, so health-probe-driven traffic routing, restarts and capacity increases are the decisions to automate first.

Sources

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Model inference server among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.