Cases1 case

1 to 1 of 1

1. IL-0413 LLM inference engines timed out or ran out of resources under load in about 60% of 156 severe incidents at Microsoft

   Microsoft2025

   Symptoms

   In Microsoft's cloud LLM inference service, the inference engines themselves failed in 94 of the 156 high-severity incidents from April to June 2025 (≈60%). Requests timed out or ran into resource exhaustion under load, and callers got HTTP 408, 429 and 5xx errors.

   Intermittent timeoutsHigh error rateHigh latency

   Trigger

   Bursty load, traffic with unusually high sampling parameters and long-running requests pushed the inference engines into resource exhaustion and timeouts.

   Root cause

   Capacity exhaustion

   Component

   Model inference server

   Contributing factor

   Monitoring blind spot

   Detection

   Across the 156 incidents, 115 (≈74%) were auto-detected, meaning health probes or DRI pipelines alerted before customers reported the problem; the others were not.

   Remediation

   • Enabled dynamic pushback at the load balancer with caps on pending tokens and requests
   • Tuned request batching and GPU/CPU replica counts
   • Improved gRPC backoff and error mapping
   • Limited sampling parameters and throttled per client
   • Enabled streaming, tuned request batching and capped pending tokens against engine connection timeouts on long-running requests
   • Increased capacity, routed traffic to healthy endpoints and restarted nodes, the actions applied mostly to inference engine failures

   Action items

   • Optimise serving for long-running requests and high sampling regimes, with connection liveness and GPU capacity-aware routing as the main levers
   • Expand health-probe-driven failover and GPU capacity-aware routing so traffic moves off degraded endpoints on its own
   • Forecast token and throughput growth and automate capacity increase decisions
   • Deploy adaptive alerting and expand engine-level telemetry to close detection gaps

   Lessons

   Hotfixes restore correctness but lengthen time to mitigate, while traffic routing and node rebalancing mitigate faster under overload and timeout patterns. Even with AIOps tooling a meaningful share of incidents still needs manual detection or judgement, so health-probe-driven traffic routing, restarts and capacity increases are the decisions to automate first.

   View case

By root cause1