Cases1 case

1 to 1 of 1

1. IL-0407 Azure OpenAI returned 5xx errors in 14 of 28 regions after an automated cleanup deleted its GPU VMs and endpoints

   Microsoft Azure2024

   Symptoms

   From 00:00 UTC on 13 July 2024 a subset of Azure OpenAI (AOAI) customers got errors, such as 5xx responses, when calling their Azure OpenAI endpoints in 14 of the 28 regions that offer the service. Service monitoring saw failure rates cross their thresholds at 00:05 and alerted engineers.

   High error rateResources deleted

   Trigger

   A change to the global configuration file of the Azure OpenAI resource automation removed 14 resource groups that were thought to be unused, and the automation then deleted every resource in them.

   Root cause

   Configuration change

   Component

   Model inference serverAutomated capacity manager

   Contributing factor

   No staged rolloutNo safeguard on destructive changeUnverified assumption

   Detection

   Service monitoring detected failure rates above the defined thresholds at 00:05 UTC and alerted engineers, five minutes after impact began.

   Remediation

   • Stopped the automation that was running the cleanup
   • Added resource locks and blocked delete requests at the ARM and Automation levels
   • Recreated the deleted resources, rebuilding the service in parallel across the 14 regions
   • Manually reconfigured Azure Redis and Private Link for GPT-4o with the Azure Redis and Azure Networking teams
   • Manually restored DALL-E, adding the managed identity between its frontend and backend to an allowlist
   • Turned the automation back on on 15 July, which restored fine-tuning deployments

   Action items

   • Make the configuration policy regional and follow Safe Deployment Practices for all later configuration updates
   • Remove incorrect metadata tags on resources the automation must not manage and exclude them in the automation configuration
   • Concentrate the regional makeup of workloads in fewer regions
   • Improve the automation that engages incident response personnel
   • Investigate additional change management controls so that all production changes go through a gated safe deployment process (estimated completion August 2024)
   • Update the communications tooling with Azure OpenAI data so affected customers can be scoped and notified faster
   • Investigate additional test coverage and verification procedures to de-risk future configuration updates (estimated completion September 2024)
   • Add active monitoring for each model family that does not depend on customer traffic
   • Build out model pools with more parallelism to shorten recovery

   Lessons

   Microsoft names the missing safe deployment step as the main gap in its change management: a configuration change delivered as a global content update could not be rolled out region by region, so a single change caused a multi-region incident. Resource groups that mixed resources the automation managed with resources it did not know had let the automation's configuration drift from what actually existed in the production subscription.

   View case

By root cause1