Incident Library
Applied filters Root causeCloud provider outageOrganisationMicrosoft AzureClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast
    Microsoft Azure2023
    Symptoms

    From about 10:30 UTC on 30 August 2023, customers had trouble accessing or using Azure, Microsoft 365 and Power Platform services in Australia East. The affected Azure services included Azure Kubernetes Service (AKS) along with storage, SQL Database, Cosmos DB, virtual machines and Azure Resource Manager.

    Control plane unresponsiveHigh error rateService outage
    Trigger

    At 08:41 UTC a lightning strike on electrical infrastructure about 18 miles away caused a utility voltage sag that shut down the cooling chillers of datacenters in one Australia East availability zone; 13 chillers failed to restart automatically and five could not be restarted, so two data halls overheated and their infrastructure was shut down.

    Root cause
    Cloud provider outage
    Component
    AKS control planeApplication database
    Contributing factor
    Single point of failureExcessive blast radius
    Detection

    Cosmos DB impact was detected by monitoring at 10:47 UTC, first automated communications went to Azure Service Health at 10:48 UTC, the public Azure Status page was updated at 11:30 UTC, and all subscriptions in Australia East were sent portal communications at 11:36 UTC.

    Remediation
    • shut down the remaining networking, compute and storage in the two affected data halls so the chilled water loop could cool down
    • restarted the five chillers manually at 12:12 UTC
    • restored power to all hardware by 15:10 UTC and brought infrastructure back in phases
    • restored AKS operations through the recovery of the SQL Database operation queue, with no other mitigation
    Action items
    • AKS: convert all operation queue SQL Database databases to be zone redundant (estimated September 2023)
    • AKS: replace all cross-region SQL Database queue usage with zone-redundant Service Bus queues (estimated September 2023)
    • Change the control timing logic of the Chiller Management System so chillers do not fail to restart after voltage fluctuations
    • Change the emergency procedure for manual restarts of simultaneous chiller failures from an adjacent sequence to a load-based sequence per data hall
    • Increase technician staffing at the datacenter as a temporary measure for manual chiller restarts
    • Storage: optimise the large-scale recovery process and update troubleshooting guides (estimated December 2023)
    • Azure Service Fabric: improve reliability of SQL Database tenant ring recovery (estimated December 2023)
    • SQL Database: review the trigger criteria of auto-failover groups (estimated October 2023)
    • SQL Database: upgrade internal tooling to enable mass migration of databases (estimated December 2023)
    • Cosmos DB: optimise Service Managed Failover for single-region-write accounts (estimated November 2023)
    • ARM: complete the migration of its storage layer to the zonally redundant architecture (estimated December 2023)
    • Incident management: harden readiness, process and playbook for power-down scenarios (estimated October 2023)
    Lessons

    A control plane spread across availability zones still fails with a zone when one of its dependencies, here the SQL Database operation queue, is not zone redundant. When a second region uses the same database, the failure crosses into that region as well. Workloads already running were not expected to be affected; what stopped was everything that needs the managed service's control plane, such as scaling and cluster changes.

    View case
Check your cluster for this

This case involves AKS control plane and Application database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Cloud provider outage1 case

Component
AKS control planeApplication database
Contributing factor
Excessive blast radiusSingle point of failure
Organisation
Microsoft Azure
  1. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast
    Microsoft Azure2023