Incident Library
Applied filters ComponentAKS control planeSymptomService outageDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast
    Microsoft Azure2023
    Symptoms

    From about 10:30 UTC on 30 August 2023, customers had trouble accessing or using Azure, Microsoft 365 and Power Platform services in Australia East. The affected Azure services included Azure Kubernetes Service (AKS) along with storage, SQL Database, Cosmos DB, virtual machines and Azure Resource Manager.

    Control plane unresponsiveHigh error rateService outage
    Trigger

    At 08:41 UTC a lightning strike on electrical infrastructure about 18 miles away caused a utility voltage sag that shut down the cooling chillers of datacenters in one Australia East availability zone; 13 chillers failed to restart automatically and five could not be restarted, so two data halls overheated and their infrastructure was shut down.

    Root cause
    Cloud provider outage
    Component
    AKS control planeDatabase
    Contributing factor
    Single point of failureExcessive blast radius
    Detection

    Cosmos DB impact was detected by monitoring at 10:47 UTC, first automated communications went to Azure Service Health at 10:48 UTC, the public Azure Status page was updated at 11:30 UTC, and all subscriptions in Australia East were sent portal communications at 11:36 UTC.

    Remediation
    • shut down the remaining networking, compute and storage in the two affected data halls so the chilled water loop could cool down
    • restarted the five chillers manually at 12:12 UTC
    • restored power to all hardware by 15:10 UTC and brought infrastructure back in phases
    • restored AKS operations through the recovery of the SQL Database operation queue, with no other mitigation
    Action items
    • AKS: convert all operation queue SQL Database databases to be zone redundant (estimated September 2023)
    • AKS: replace all cross-region SQL Database queue usage with zone-redundant Service Bus queues (estimated September 2023)
    • Change the control timing logic of the Chiller Management System so chillers do not fail to restart after voltage fluctuations
    • Change the emergency procedure for manual restarts of simultaneous chiller failures from an adjacent sequence to a load-based sequence per data hall
    • Increase technician staffing at the datacenter as a temporary measure for manual chiller restarts
    • Storage: optimise the large-scale recovery process and update troubleshooting guides (estimated December 2023)
    • Azure Service Fabric: improve reliability of SQL Database tenant ring recovery (estimated December 2023)
    • SQL Database: review the trigger criteria of auto-failover groups (estimated October 2023)
    • SQL Database: upgrade internal tooling to enable mass migration of databases (estimated December 2023)
    • Cosmos DB: optimise Service Managed Failover for single-region-write accounts (estimated November 2023)
    • ARM: complete the migration of its storage layer to the zonally redundant architecture (estimated December 2023)
    • Incident management: harden readiness, process and playbook for power-down scenarios (estimated October 2023)
    Lessons

    A control plane spread across availability zones still fails with a zone when one of its dependencies, here the SQL Database operation queue, is not zone redundant. When a second region uses the same database, the failure crosses into that region as well. Workloads already running were not expected to be affected; what stopped was everything that needs the managed service's control plane, such as scaling and cluster changes.

    View case
  2. IL-0469 The managed Azure Kubernetes API server for the data100 hub stopped answering, so the stopped hub pod was not recreated
    UC Berkeley2019
    Symptoms

    On 25 February 2019 the Kubernetes API server of the managed Azure cluster running the data100 hub became unreachable. kubectl failed for every operation and requests to create new resources failed, so when the hub pod stopped, no new one was created and users saw a proxy error message.

    Control plane unresponsiveService outage
    Root cause
    Cloud provider outage
    Component
    AKS control planeKubernetes API serverkubelet
    Contributing factor
    Limited debug access
    Detection

    Infrastructure staff were notified in Slack at 11:34 and found that kubectl failed for all operations while the hub proxy was up.

    Remediation
    • Created a new cluster and requested a new NFS server in its resource group
    • Detached the data volumes from the old NFS server and moved them to the new cluster's resource group
    • Imported the ZFS pool into the new NFS server
    • Created the staging hub with helm and updated DNS
    • Created the prod hub with helm and updated DNS
    Action items
    • Monitor the API server endpoint remotely and alert when it is down
    View case
Check your cluster for this

These 2 cases involve AKS control plane and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Cloud provider outage2 cases

Component
AKS control plane2Database1Kubernetes API server1kubelet1
Contributing factor
Excessive blast radius1Limited debug access1Single point of failure1
Organisation
Microsoft Azure1UC Berkeley1
  1. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast
    Microsoft Azure2023
  2. IL-0469 The managed Azure Kubernetes API server for the data100 hub stopped answering, so the stopped hub pod was not recreated
    UC Berkeley2019