Incident Library

IL-0407 Microsoft Azure · 2024-07-13azureai-infra

Azure OpenAI returned 5xx errors in 14 of 28 regions after an automated cleanup deleted its GPU VMs and endpoints

Azure OpenAI uses an internal automation service to create and manage the Azure resources it needs: Azure GPU virtual machines that host the OpenAI large language models, and Azure Machine Learning workspaces whose managed online endpoints serve inference requests. The same automation orchestrates fine-tuned model deployments. The automation runs per region, but one global configuration file in a code repository defines the resources Azure OpenAI needs in every region. As the service grew very fast, several components started to use the same resource structure, and some resource groups came to hold both resources the automation managed and resources it did not know about. The configuration file listed 14 resource groups as holding only resources the service no longer needed, while they in fact held GPU compute VMs and model endpoints serving both the Pay-As-You-Go (PayGo) and Processing Throughput Unit (PTU) offers.

During ongoing work to clean up unused resources, to stay within subscription limits and shrink the security attack surface, a change removed those 14 resource groups from the global configuration file. The change took effect soon after it was committed, and everything in the groups was deleted within a short time. Because the change shipped as a global content update, no safe deployment process gated it region by region, which made the incident multi-region. Engineers began stopping the automation at 01:00; the cleanup job stopped on its own at 01:40 without deleting everything, and at 02:00 resource locks were added and delete requests were blocked at the ARM and Automation levels.

Rebuilding started at 02:00 in parallel across the 14 regions. The models are very large and can only be moved over secure channels, and inside each model pool (one per region per model) deployments run one after another, so copying time was a large factor. Most models and regions had recovered by 10:20 and all base models by 23:35 UTC. GPT-4o needed manual reconfiguration of Azure Redis and Private Link with the Azure Redis and Azure Networking teams, and DALL-E needed manual work to add the managed identity between its frontend and backend to an allowlist. The automation that creates fine-tuning deployments was kept off over the weekend so it could not delete anything again; when it was turned back on on 15 July, fine-tuned deployments were restored by 16:54 UTC. The first Azure Status page post went out at 02:22, delayed by difficulty scoping the affected customers and gaps in the communications tooling. No retrieval-augmented generation (RAG) or fine-tuning model data was lost.

Severity
–
Scope
Azure OpenAI endpoints in 14 of 28 regions returned errors such as 5xx until 23:35 UTC on 13 July; new Standard fine-tuned model deployments unavailable in four regions until 16:54 UTC on 15 July
Users affected
A subset of Azure OpenAI customers
Region
Australia East, Brazil South, Canada Central, Canada East, East US 2, France Central, Korea Central, North Central US, Norway East, Poland Central, South Africa North, South India, Sweden Central, UK South; fine-tuning also Switzerland West
Time to detect
5m
Time to resolve
2d 16h 54m

Symptoms

From 00:00 UTC on 13 July 2024 a subset of Azure OpenAI (AOAI) customers got errors, such as 5xx responses, when calling their Azure OpenAI endpoints in 14 of the 28 regions that offer the service. Service monitoring saw failure rates cross their thresholds at 00:05 and alerted engineers.

The affected regions were Australia East, Brazil South, Canada Central, Canada East, East US 2, France Central, Korea Central, North Central US, Norway East, Poland Central, South Africa North, South India, Sweden Central and UK South. Impact grew as more regions became unhealthy: the deletion took 30 to 40 minutes to complete, so each region, and even each endpoint within a region, went offline at a different time.

By 00:30 engineers had correlated alerts, service metrics and telemetry dashboards and knew a multi-region issue was unfolding. At 00:45 they saw that backend endpoints were being deleted but not how or why, and on-call engineers followed two hypotheses. At 01:00 they identified the automated cleanup operation that was deleting the resources.

In four regions, East US 2, North Central US, Sweden Central and Switzerland West, new Standard (Pay-As-You-Go) fine-tuned model deployments were unavailable from 01:00 UTC on 13 July to 16:54 UTC on 15 July, and deleting a deployment might not take effect, while inference on models already deployed there kept working.

Symptom High error rateResources deleted

Root cause

Root cause Configuration change

Component Model inference serverAutomated capacity manager
Contributing factor No staged rolloutNo safeguard on destructive changeUnverified assumption
Trigger
A change to the global configuration file of the Azure OpenAI resource automation removed 14 resource groups that were thought to be unused, and the automation then deleted every resource in them.

Detection

Service monitoring detected failure rates above the defined thresholds at 00:05 UTC and alerted engineers, five minutes after impact began.

Remediation

Action items

Lessons

Microsoft names the missing safe deployment step as the main gap in its change management: a configuration change delivered as a global content update could not be rolled out region by region, so a single change caused a multi-region incident. Resource groups that mixed resources the automation managed with resources it did not know had let the automation's configuration drift from what actually existed in the production subscription.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Model inference server, Automated capacity manager among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.