Cases24 cases

14 to 18 of 24

1. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast

   Microsoft Azure2023

   Symptoms

   From about 10:30 UTC on 30 August 2023, customers had trouble accessing or using Azure, Microsoft 365 and Power Platform services in Australia East. The affected Azure services included Azure Kubernetes Service (AKS) along with storage, SQL Database, Cosmos DB, virtual machines and Azure Resource Manager.

   Control plane unresponsiveHigh error rateService outage

   Trigger

   At 08:41 UTC a lightning strike on electrical infrastructure about 18 miles away caused a utility voltage sag that shut down the cooling chillers of datacenters in one Australia East availability zone; 13 chillers failed to restart automatically and five could not be restarted, so two data halls overheated and their infrastructure was shut down.

   Root cause

   Cloud provider outage

   Component

   AKS control planeDatabase

   Contributing factor

   Single point of failureExcessive blast radius

   Detection

   Cosmos DB impact was detected by monitoring at 10:47 UTC, first automated communications went to Azure Service Health at 10:48 UTC, the public Azure Status page was updated at 11:30 UTC, and all subscriptions in Australia East were sent portal communications at 11:36 UTC.

   Remediation

   • shut down the remaining networking, compute and storage in the two affected data halls so the chilled water loop could cool down
   • restarted the five chillers manually at 12:12 UTC
   • restored power to all hardware by 15:10 UTC and brought infrastructure back in phases
   • restored AKS operations through the recovery of the SQL Database operation queue, with no other mitigation

   Action items

   • AKS: convert all operation queue SQL Database databases to be zone redundant (estimated September 2023)
   • AKS: replace all cross-region SQL Database queue usage with zone-redundant Service Bus queues (estimated September 2023)
   • Change the control timing logic of the Chiller Management System so chillers do not fail to restart after voltage fluctuations
   • Change the emergency procedure for manual restarts of simultaneous chiller failures from an adjacent sequence to a load-based sequence per data hall
   • Increase technician staffing at the datacenter as a temporary measure for manual chiller restarts
   • Storage: optimise the large-scale recovery process and update troubleshooting guides (estimated December 2023)
   • Azure Service Fabric: improve reliability of SQL Database tenant ring recovery (estimated December 2023)
   • SQL Database: review the trigger criteria of auto-failover groups (estimated October 2023)
   • SQL Database: upgrade internal tooling to enable mass migration of databases (estimated December 2023)
   • Cosmos DB: optimise Service Managed Failover for single-region-write accounts (estimated November 2023)
   • ARM: complete the migration of its storage layer to the zonally redundant architecture (estimated December 2023)
   • Incident management: harden readiness, process and playbook for power-down scenarios (estimated October 2023)

   Lessons

   A control plane spread across availability zones still fails with a zone when one of its dependencies, here the SQL Database operation queue, is not zone redundant. When a second region uses the same database, the failure crosses into that region as well. Workloads already running were not expected to be affected; what stopped was everything that needs the managed service's control plane, such as scaling and cluster changes.

   View case
2. IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all traffic

   GitLab2022

   Symptoms

   GitLab's monitoring and alerting fired at once. One of the alerts reported a Patroni failover, and users of GitLab.com received 5XX errors; the CDN counted about 13 million requests answered with 503 during the outage.

   Service outageHigh error rateResources deleted

   Trigger

   An engineer ran a routine task with GitLab's tools in a local development environment; during a local deploy or rollback operation the tool's context changed from local development to production, and the change was applied to production.

   Root cause

   Operator error

   Component

   Database

   Contributing factor

   No safeguard on destructive change

   Detection

   GitLab's monitoring and alerting fired immediately; one of the alerts reported a Patroni failover.

   Remediation

   • re-applied the Consul release to the production Kubernetes cluster, with manual steps because the tooling did not support the redeploy directly

   Action items

   • Improve the Kubernetes deployment process to reduce the risk of running harmful deployment operations against production from local environments (https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16875)
   • Integrate Teleport with the Kubernetes clusters, which could give SREs a read-only role by default (https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16221)

   Lessons

   A tool that can switch its target from a local environment to production during a deploy or rollback turns routine local work into a production change. GitLab plans to improve its Kubernetes deployment process so that harmful deployment operations are less likely to be run against production from local environments, and to integrate Teleport with its clusters, which could give SREs a read-only role by default. Detection and diagnosis were fast because the alerts led straight from Patroni to its dependency, Consul. Bringing the deleted release back needed manual work, because the tooling did not make the redeploy straightforward, but it was still done relatively quickly.

   View case
3. IL-0500 UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up

   2i2c2022

   Symptoms

   On 31 October 2022 at 6:27 PM Pacific time, users of the UToronto JupyterHub run by 2i2c could no longer log in or start their servers, and the hub answered with 500 errors. Users who were already logged in could keep using the hub.

   High error ratePersistent volume full

   Trigger

   The JupyterHub log file, written to the same 10G disk as the hub's sqlite database, grew until the disk was completely full.

   Root cause

   Capacity exhaustion

   Component

   DatabasePersistent volumes and stateful setsJupyterHub

   Contributing factor

   No alert on limit

   Detection

   A user emailed 2i2c support at 7:13 PM that users could not log in; the team acknowledged the issue at 8:33 PM. There was no alert for the problem.

   Remediation

   • Compressed the hub log file into /tmp from a shell in the hub pod, because it could not be compressed in place on the full disk
   • Copied the compressed log to a local machine so the hub pod could be restarted without losing it
   • Resized the hub database disk (the hub-db-dir PVC) from 10G to 60G
   • Restarted the hub pod, which came back with the larger disk and restored service

   Action items

   • Stop keeping hub logs on the same disk as the hub database
   • Add a Grafana graph of free space on the JupyterHub database disk
   • Alert on elevated error rates

   Lessons

   Kubernetes dynamic volume resizing let the hub database disk grow from 10G to 60G without much effort. The risk of keeping logs on the database disk had been identified during the earlier migration, but the team had no solid process for following it through, and because no alert existed the team had to wait for a person to report the outage.

   View case
4. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary

   GitLab2022

   Symptoms

   On 2022-04-25 the Sidekiq Apdex (the share of background jobs that finish within a target time) dropped, and an engineer declared an incident in Slack at 15:13 UTC. Users saw intermittent slowness across GitLab.com, and asynchronous work such as pipeline creation and import and export took unusually long.

   High latencyDatabase overloadOOMKilledHigh error rate

   Trigger

   The Consul agent in GitLab.com's GKE pods had memory demand spikes that did not fit in the pods' 500MB memory limit, and the agent was killed again and again.

   Root cause

   Capacity exhaustion

   Component

   Container memory limitService discoveryDatabase

   Contributing factor

   –

   Detection

   An engineer declared the incident in Slack at 15:13 UTC after the Sidekiq Apdex dropped; the second event was found when monitoring detected an elevated error rate at 01:21 UTC on 2022-04-26.

   Remediation

   • planned at 16:13 to raise the per-user connection limit for the gitlab database user and the pgbouncer backend pools for the main database; changes were applied at 17:26 and confirmed at 17:47
   • raised the pod memory limits from 500MB to 1GB so the Consul agents survive memory spikes

   Action items

   • Monitor the Consul agents on GKE pods to see whether the 1GB memory limit keeps them healthy
   • Investigate the Consul memory demand spikes and what can be done beyond raising the hard memory limit

   Lessons

   When the local Consul agent that Rails uses for database service discovery keeps dying, Rails does not fail; it sends every query to the last known primary. What engineers saw was database saturation and slow background jobs, which pointed the investigation first at a recent pgbouncer pool change. The memory limit of the Consul agent in every pod therefore decided the load on the primary database.

   View case
5. IL-0462 Some MySQL pods on Shopify's GKE clusters took up to 30 minutes to start, slowing instance replacement

   Shopify2021

   Symptoms

   In 2021 Shopify's KateSQL, a database-as-a-service platform that runs several hundred production MySQL instances on GKE, found that some MySQL pods started and shut down much more slowly than similar pods with the same data set. Instance replacement, the procedure that creates a new MySQL replica pod and stops the old one, showed pod creation times ranging from 10 to 30 minutes, which held up maintenance such as configuration changes and upgrades.

   Slow pod startup or shutdownHigh latency

   Root cause

   Software defect

   Component

   Linux kernelNode operating system imageDatabase

   Contributing factor

   Unpatched known bug

   Detection

   KateSQL's instance replacement operations showed inconsistent MySQL pod creation times, and the team then measured the mysql-init container and MySQL startup times across pods.

   Remediation

   • Cordoned older nodes, moved MySQL pods to new nodes through instance replacement and drained the old nodes
   • Forced the kubepods memory cgroup to empty its dead cgroups on an affected node to test the fix
   • Created new GKE 1.18 clusters and started creating MySQL pods on them

   Lessons

   The investigation was one of the longest the Database Platform group had carried out, because of the nature of the problem, how hard it was to reproduce and the lack of an established method to follow; once the cause was known, there were several ways to fix it.

   View case

By root cause7