IL-0235 Microsoft Azure · 2026-03-09ai-infraazure

Azure OpenAI GPT-5.2 returned HTTP 400 in seven regions after a configuration update the model engine did not support

Azure OpenAI Service processes customer requests on model engines deployed in many Azure regions, with traffic routing systems deciding where each request goes; depending on the deployment type (Global, Data Zone or Regional), a request may be routed to other regions within a defined geography. An update to the GPT-5.2 model introduced a configuration change that the deployed engine code did not support, so requests routed to engines with the mismatch could not be processed and failed with HTTP 400. The update followed Safe Deployment Practices, which roll out region by region in stages, but the early stages had too few backend model instances for the problem to show, and the rollout had reached the whole fleet before customer impact was recognised. Customer impact began at 23:20 UTC on 9 March and was detected at 00:19 UTC on 10 March, after which teams began investigating and building a hotfix.

At 03:18 UTC the team decided that a rollback would mitigate faster than a hotfix and began rolling back the impacted model. During recovery an unrelated problem in the internal capacity telemetry fed incomplete regional capacity data to routing, which sent too much traffic to a limited set of regions and kept intermittent HTTP 429 errors going; this was identified at 10:55 UTC. The rollback finished in all affected regions at 18:00 UTC, full capacity information was restored and traffic distribution returned to normal at 19:30 UTC, and the service was declared fully restored at 19:32 UTC on 10 March 2026.

Severity: –
Scope: Azure OpenAI Service requests to the GPT-5.2 model returned HTTP 400 and HTTP 429 errors; other GPT models were unaffected
Users affected: –
Region: Australia East, Central US, East US 2, Korea Central, Norway East, Sweden Central and UK South
Time to detect: 59m
Time to resolve: 20h 12m

Symptoms

From 23:20 UTC on 9 March 2026 Azure OpenAI Service customers received HTTP 400 and HTTP 429 error responses for the GPT-5.2 model in Australia East, Central US, East US 2, Korea Central, Norway East, Sweden Central and UK South. All other GPT models kept working.

The HTTP 400 errors went unnoticed for about an hour. HTTP 400 responses occur in normal use, and the service monitoring watched only for client-side errors caused by bad parameters in user requests, not for anomalies on the service side. The issue was detected through service monitoring at 00:19 UTC on 10 March.

While the rollback was under way, HTTP 429 errors continued intermittently for some customers. An unrelated fault in the internal telemetry that reports real-time capacity per region left the traffic routing system with incomplete capacity data, so a disproportionate share of traffic went to a few available regions and put them under resource pressure, while other regions could have taken requests. This was found at 10:55 UTC and the resource constraints were investigated from 12:40 UTC.

Symptom High error rate Load imbalance

Root cause

Root cause Configuration change

Component Model inference server Model request routing

Contributing factor Unrepresentative staging Monitoring blind spot Overlapping failures

Trigger: An update to the Azure OpenAI GPT-5.2 model enabled feature settings meant to improve service efficiency and resilience. The model engine version running in production did not yet support those settings. The update was rolled out to the whole fleet in stages under Safe Deployment Practices.

Detection

Service monitoring detected the issue at 00:19 UTC on 10 March, about an hour after impact began. Detection was late because monitoring was set up for client-side HTTP 400 errors and not for service-side anomalies.

Remediation

Rolled back the GPT-5.2 configuration in all affected regions instead of waiting for a hotfix
Completed the routing updates and restored full regional capacity information so traffic was spread normally again

Action items

Train engineers on the operating procedures, including which scenarios can be rolled back quickly
Store a cached known-good version of the traffic routing details for when the capacity service cannot provide current data
Set thresholds for service-side HTTP 400 errors in monitoring
Improve anomaly detection on HTTP 4xx errors to alert on unusual error rates below the usual thresholds
Add signals to the deployment systems so problematic rollouts stop automatically
Make sure early rollout stages have enough backend model instances to catch such problems

Lessons

–

Sources

https://azure.status.microsoft/en-us/status/history/?trackingId=8GCS-858 · official-status

References

Related cases

IL-0407 Azure OpenAI returned 5xx errors in 14 of 28 regions after an automated cleanup deleted its GPU VMs and endpointsMicrosoft Azure · 2024
same root cause · shared Model inference server

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Model inference server, Model request routing among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0235
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0235/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.