Incident Library
Applied filters Root causeConfiguration changeComponentAutomated capacity managerFactorNo safeguard on destructive changeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0407 Azure OpenAI returned 5xx errors in 14 of 28 regions after an automated cleanup deleted its GPU VMs and endpoints
    Microsoft Azure2024
    Symptoms

    From 00:00 UTC on 13 July 2024 a subset of Azure OpenAI (AOAI) customers got errors, such as 5xx responses, when calling their Azure OpenAI endpoints in 14 of the 28 regions that offer the service. Service monitoring saw failure rates cross their thresholds at 00:05 and alerted engineers.

    High error rateResources deleted
    Trigger

    A change to the global configuration file of the Azure OpenAI resource automation removed 14 resource groups that were thought to be unused, and the automation then deleted every resource in them.

    Root cause
    Configuration change
    Component
    Model inference serverAutomated capacity manager
    Contributing factor
    No staged rolloutNo safeguard on destructive changeUnverified assumption
    Detection

    Service monitoring detected failure rates above the defined thresholds at 00:05 UTC and alerted engineers, five minutes after impact began.

    Remediation
    • Stopped the automation that was running the cleanup
    • Added resource locks and blocked delete requests at the ARM and Automation levels
    • Recreated the deleted resources, rebuilding the service in parallel across the 14 regions
    • Manually reconfigured Azure Redis and Private Link for GPT-4o with the Azure Redis and Azure Networking teams
    • Manually restored DALL-E, adding the managed identity between its frontend and backend to an allowlist
    • Turned the automation back on on 15 July, which restored fine-tuning deployments
    Action items
    • Make the configuration policy regional and follow Safe Deployment Practices for all later configuration updates
    • Remove incorrect metadata tags on resources the automation must not manage and exclude them in the automation configuration
    • Concentrate the regional makeup of workloads in fewer regions
    • Improve the automation that engages incident response personnel
    • Investigate additional change management controls so that all production changes go through a gated safe deployment process (estimated completion August 2024)
    • Update the communications tooling with Azure OpenAI data so affected customers can be scoped and notified faster
    • Investigate additional test coverage and verification procedures to de-risk future configuration updates (estimated completion September 2024)
    • Add active monitoring for each model family that does not depend on customer traffic
    • Build out model pools with more parallelism to shorten recovery
    Lessons

    Microsoft names the missing safe deployment step as the main gap in its change management: a configuration change delivered as a global content update could not be rolled out region by region, so a single change caused a multi-region incident. Resource groups that mixed resources the automation managed with resources it did not know had let the automation's configuration drift from what actually existed in the production subscription.

    View case
Check your cluster for this

This case involves Automated capacity manager and Model inference server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
Automated capacity managerModel inference server
Contributing factor
Unverified assumptionNo safeguard on destructive changeNo staged rollout
Organisation
Microsoft Azure
  1. IL-0407 Azure OpenAI returned 5xx errors in 14 of 28 regions after an automated cleanup deleted its GPU VMs and endpoints
    Microsoft Azure2024