Incident Library

Applied filters Root causeConfiguration changeComponentModel inference serverSymptomHigh error rateClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

Cases2 cases

1 to 2 of 2

IL-0235 Azure OpenAI GPT-5.2 returned HTTP 400 in seven regions after a configuration update the model engine did not support
Microsoft Azure2026
Symptoms
From 23:20 UTC on 9 March 2026 Azure OpenAI Service customers received HTTP 400 and HTTP 429 error responses for the GPT-5.2 model in Australia East, Central US, East US 2, Korea Central, Norway East, Sweden Central and UK South. All other GPT models kept working.
High error rate Load imbalance
Trigger
An update to the Azure OpenAI GPT-5.2 model enabled feature settings meant to improve service efficiency and resilience. The model engine version running in production did not yet support those settings. The update was rolled out to the whole fleet in stages under Safe Deployment Practices.
Root cause
Configuration change
Component
Model inference server Model request routing
Contributing factor
Unrepresentative staging Monitoring blind spot Overlapping failures
Detection
Service monitoring detected the issue at 00:19 UTC on 10 March, about an hour after impact began. Detection was late because monitoring was set up for client-side HTTP 400 errors and not for service-side anomalies.
Remediation
Rolled back the GPT-5.2 configuration in all affected regions instead of waiting for a hotfix
Completed the routing updates and restored full regional capacity information so traffic was spread normally again
Action items
Train engineers on the operating procedures, including which scenarios can be rolled back quickly
Store a cached known-good version of the traffic routing details for when the capacity service cannot provide current data
Set thresholds for service-side HTTP 400 errors in monitoring
Improve anomaly detection on HTTP 4xx errors to alert on unusual error rates below the usual thresholds
Add signals to the deployment systems so problematic rollouts stop automatically
Make sure early rollout stages have enough backend model instances to catch such problems
View case
IL-0407 Azure OpenAI returned 5xx errors in 14 of 28 regions after an automated cleanup deleted its GPU VMs and endpoints
Microsoft Azure2024
Symptoms
From 00:00 UTC on 13 July 2024 a subset of Azure OpenAI (AOAI) customers got errors, such as 5xx responses, when calling their Azure OpenAI endpoints in 14 of the 28 regions that offer the service. Service monitoring saw failure rates cross their thresholds at 00:05 and alerted engineers.
High error rate Resources deleted
Trigger
A change to the global configuration file of the Azure OpenAI resource automation removed 14 resource groups that were thought to be unused, and the automation then deleted every resource in them.
Root cause
Configuration change
Component
Model inference server Automated capacity manager
Contributing factor
No staged rollout No safeguard on destructive change Unverified assumption
Detection
Service monitoring detected failure rates above the defined thresholds at 00:05 UTC and alerted engineers, five minutes after impact began.
Remediation
Stopped the automation that was running the cleanup
Added resource locks and blocked delete requests at the ARM and Automation levels
Recreated the deleted resources, rebuilding the service in parallel across the 14 regions
Manually reconfigured Azure Redis and Private Link for GPT-4o with the Azure Redis and Azure Networking teams
Manually restored DALL-E, adding the managed identity between its frontend and backend to an allowlist
Turned the automation back on on 15 July, which restored fine-tuning deployments
Action items
Make the configuration policy regional and follow Safe Deployment Practices for all later configuration updates
Remove incorrect metadata tags on resources the automation must not manage and exclude them in the automation configuration
Concentrate the regional makeup of workloads in fewer regions
Improve the automation that engages incident response personnel
Investigate additional change management controls so that all production changes go through a gated safe deployment process (estimated completion August 2024)
Update the communications tooling with Azure OpenAI data so affected customers can be scoped and notified faster
Investigate additional test coverage and verification procedures to de-risk future configuration updates (estimated completion September 2024)
Add active monitoring for each model family that does not depend on customer traffic
Build out model pools with more parallelism to shorten recovery
Lessons
Microsoft names the missing safe deployment step as the main gap in its change management: a configuration change delivered as a global content update could not be rolled out region by region, so a single change caused a multi-region incident. Resource groups that mixed resources the automation managed with resources it did not know had let the automation's configuration drift from what actually existed in the production subscription.
View case

Check your cluster for this

These 2 cases involve Model inference server and Automated capacity manager. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change2 cases

Component: Model inference server2 Automated capacity manager1 Model request routing1
Contributing factor: Unverified assumption1 Monitoring blind spot1 No safeguard on destructive change1 No staged rollout1 Overlapping failures1 Unrepresentative staging1
Organisation: Microsoft Azure2