Cases2 cases

1 to 2 of 2

1. IL-0235 Azure OpenAI GPT-5.2 returned HTTP 400 in seven regions after a configuration update the model engine did not support

   Microsoft Azure2026

   Symptoms

   From 23:20 UTC on 9 March 2026 Azure OpenAI Service customers received HTTP 400 and HTTP 429 error responses for the GPT-5.2 model in Australia East, Central US, East US 2, Korea Central, Norway East, Sweden Central and UK South. All other GPT models kept working.

   High error rateLoad imbalance

   Trigger

   An update to the Azure OpenAI GPT-5.2 model enabled feature settings meant to improve service efficiency and resilience. The model engine version running in production did not yet support those settings. The update was rolled out to the whole fleet in stages under Safe Deployment Practices.

   Root cause

   Configuration change

   Component

   Model inference serverModel request routing

   Contributing factor

   Unrepresentative stagingMonitoring blind spotOverlapping failures

   Detection

   Service monitoring detected the issue at 00:19 UTC on 10 March, about an hour after impact began. Detection was late because monitoring was set up for client-side HTTP 400 errors and not for service-side anomalies.

   Remediation

   • Rolled back the GPT-5.2 configuration in all affected regions instead of waiting for a hotfix
   • Completed the routing updates and restored full regional capacity information so traffic was spread normally again

   Action items

   • Train engineers on the operating procedures, including which scenarios can be rolled back quickly
   • Store a cached known-good version of the traffic routing details for when the capacity service cannot provide current data
   • Set thresholds for service-side HTTP 400 errors in monitoring
   • Improve anomaly detection on HTTP 4xx errors to alert on unusual error rates below the usual thresholds
   • Add signals to the deployment systems so problematic rollouts stop automatically
   • Make sure early rollout stages have enough backend model instances to catch such problems

   View case
2. IL-0413 LLM inference engines timed out or ran out of resources under load in about 60% of 156 severe incidents at Microsoft

   Microsoft2025

   Symptoms

   In Microsoft's cloud LLM inference service, the inference engines themselves failed in 94 of the 156 high-severity incidents from April to June 2025 (≈60%). Requests timed out or ran into resource exhaustion under load, and callers got HTTP 408, 429 and 5xx errors.

   Intermittent timeoutsHigh error rateHigh latency

   Trigger

   Bursty load, traffic with unusually high sampling parameters and long-running requests pushed the inference engines into resource exhaustion and timeouts.

   Root cause

   Capacity exhaustion

   Component

   Model inference server

   Contributing factor

   Monitoring blind spot

   Detection

   Across the 156 incidents, 115 (≈74%) were auto-detected, meaning health probes or DRI pipelines alerted before customers reported the problem; the others were not.

   Remediation

   • Enabled dynamic pushback at the load balancer with caps on pending tokens and requests
   • Tuned request batching and GPU/CPU replica counts
   • Improved gRPC backoff and error mapping
   • Limited sampling parameters and throttled per client
   • Enabled streaming, tuned request batching and capped pending tokens against engine connection timeouts on long-running requests
   • Increased capacity, routed traffic to healthy endpoints and restarted nodes, the actions applied mostly to inference engine failures

   Action items

   • Optimise serving for long-running requests and high sampling regimes, with connection liveness and GPU capacity-aware routing as the main levers
   • Expand health-probe-driven failover and GPU capacity-aware routing so traffic moves off degraded endpoints on its own
   • Forecast token and throughput growth and automate capacity increase decisions
   • Deploy adaptive alerting and expand engine-level telemetry to close detection gaps

   Lessons

   Hotfixes restore correctness but lengthen time to mitigate, while traffic routing and node rebalancing mitigate faster under overload and timeout patterns. Even with AIOps tooling a meaningful share of incidents still needs manual detection or judgement, so health-probe-driven traffic routing, restarts and capacity increases are the decisions to automate first.

   View case

By root cause2