Cases6 cases

1 to 5 of 6

1. IL-0367 Azure Storage in Central US rejected VM disk I/O, taking down VMs and the SQL Database and Cosmos DB services on them

   Microsoft Azure2024

   Symptoms

   From 21:40 UTC on 18 July 2024, virtual machines in the Azure Central US region lost access to their disks, and the services running on those VMs failed. Azure SQL Database and SQL Managed Instance showed unhealthy nodes and login failures, Cosmos DB requests failed, and service management operations failed across many Azure services.

   Storage disconnected or stale mountsService outageHigh error rateFailover failureStale endpoint routing

   Trigger

   Routine changes to the VM host fleet led to a new allow list of VM host address ranges being generated and published to all Storage scale units in Central US. Recent network configuration changes had made some of the infrastructure file servers that hold the source data unreachable from the server generating the list, so the published list was incomplete.

   Root cause

   Configuration change

   Component

   Cloud block storageDatabaseDatabase failover manager

   Contributing factor

   Unsafe failure behaviourNo staged rolloutExcessive blast radiusMonitoring blind spot

   Detection

   Service monitoring detected the drop in VM availability at 22:06 UTC while the allow list update was being published. SQL monitoring detected unhealthy nodes and login failures within one minute of SQL becoming unavailable, and monitoring flagged the availability drop for Cosmos DB, SQL DB and SQL DB Managed Instance at 22:14 UTC.

   Remediation

   • Halted the allow list update workflow worldwide
   • Paused all deployments in the Central US region
   • Paused compute buildout deployments in all regions
   • Updated the configuration of all Storage scale units in Central US, restoring storage availability by 02:55 UTC
   • Started geo-failovers for SQL databases with automatic failover policies
   • Restarted SQL gateway nodes to refresh their caches and route connections to the right nodes
   • Forced completion of SQL failovers that had not completed
   • Failed over the Cosmos DB control plane and active-passive accounts with a write region in Central US
   • Cancelled customer-triggered graceful switch region operations that blocked Cosmos DB failovers
   • Removed locks on Cosmos DB service metadata to force-complete stuck failovers
   • Repaired impacted Cosmos DB partitions for single-region accounts
   • Repaired backend nodes and failed Cosmos DB accounts back to Central US, redoing failback where it failed

   Action items

   • Make the allow list generation workflow detect incomplete source information and halt
   • Alert on storage requests rejected by allow list checks
   • Deploy allow lists one Availability Zone and storage type at a time, over 24 hours
   • Add VM health checks and an automatic stop to the allow list deployment workflow
   • Reevaluate the policy for Microsoft-managed failover of SQL failover groups
   • Fix the Cosmos DB failback workflow that broke some MongoDB drivers for MongoDB API accounts
   • Fix Cosmos DB failback for accounts with private endpoints so it does not need redoing
   • Make the storage data-plane firewall reject invalid allow lists and keep the last-known-good state
   • Improve Azure NetApp Files monitors so impacting events are detected and classified in time
   • Monitor more Azure NetApp Files service metrics to correlate similar events faster
   • Make the SQL Service Fabric cluster location change notifications reliable under load
   • Make SQL geo-failover workflows more robust so they complete
   • Remove the issues that delayed 5% of Cosmos DB failovers
   • Migrate Azure DevOps customer metadata to the right geography
   • Decouple Azure NetApp Files regional reads and writes to keep this class of issue to one region
   • Evaluate caching to reduce Azure NetApp Files reliance on stored utilization data
   • Add automatic per-partition failover for Cosmos DB multi-region active-passive accounts
   • Build the Resilient Ephemeral OS disk improvement so VMs survive storage incidents better

   View case
2. IL-0295 Pusher's Channels publish API on the mt1 cluster was largely unavailable for 11 minutes while a Redis master failed over

   Pusher2022

   Symptoms

   At 10:42 UTC on 11 April 2022, the API for publishing messages to Pusher Channels on the mt1 cluster became largely unavailable, and from about 10:44 requests to the MT1 API returned more errors.

   Service outageHigh latencySplit brain

   Trigger

   An engineer restarted the master node of an mt1 Redis group to apply a larger replication buffer, after restarting both replicas of that group.

   Root cause

   Software defect

   Component

   In-memory cacheDatabase failover manager

   Contributing factor

   Unverified assumptionUnrepresentative staging

   Detection

   Pusher saw error rates rise for requests to the MT1 API from about 10:44 UTC.

   Action items

   • Fix the Redis shutdown script to check that failover has completed and the node is no longer a master before terminating

   View case
3. IL-0289 A failing SSD made updown.io's primary MongoDB about 200 times slower, no failover happened and the site went down

   updown.io2019

   Symptoms

   At 2:30 am CEST on 3 April 2019, the primary MongoDB server of updown.io started answering queries about 200 times slower than usual, and the website and the monitoring daemons went almost completely down.

   High latencyService outageFailover failure

   Trigger

   One of the three SSDs in the primary MongoDB server's RAID5 array began to fail, slowing the server's IO without taking the server down.

   Root cause

   Hardware failure

   Component

   DatabaseDatabase failover manager

   Contributing factor

   –

   Detection

   The founder, who was asleep when the slowdown began, noticed the problem at 8:45 am CEST and found the IO performance issue by inspecting the primary server.

   Remediation

   • Forced the primary to step down after making sure the secondary was ready to take over
   • Took the failing SSD out of the RAID array, which restored the old primary's performance
   • Asked the hosting provider to replace the SSD
   • Replaced the less powerful secondary with a more powerful server

   Action items

   • Write a script that detects when MongoDB is up but not responsive and forces a step-down automatically
   • Shut down and delete the old secondary once nothing is left to investigate on it
   • Make sure the new secondary can run as primary for several days
   • Test the automatic step-down script once the primary server is healthy again
   • Provide a proper status page for updown.io
   • Replace the custom script with MongoDB's Storage Node Watchdog

   View case
4. IL-0219 GoCardless's API and Dashboard failed every request for 1 hour 50 minutes when Pacemaker did not promote a replica

   GoCardless2017

   Symptoms

   At 15:09 BST on 10 October 2017, GoCardless's monitoring detected a total outage of its API and Dashboard: every request failed and returned an error. By 15:11 engineers saw evidence of a disk array failure on the primary Postgres node, but the Pacemaker cluster had not failed over to the synchronous replica as it should have.

   Service outageFailover failure

   Trigger

   The RAID controller of the primary Postgres node lost 3 disks of its array at once, so all reads and writes on the primary failed.

   Root cause

   Hardware failure

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Overlapping failuresUnverified assumption

   Detection

   GoCardless's monitoring detected the total outage of the API and Dashboard at 15:09 BST, and engineers began investigating.

   Remediation

   • Powered off the broken primary Postgres node
   • Cleared the Pacemaker error counts with crm resource cleanup, which had no effect
   • Edited recovery.conf on the synchronous replica under maintenance mode several times, which Pacemaker reverted
   • Put the cluster into maintenance mode, configured the synchronous replica as primary and started Postgres by hand
   • Reconfigured the backend applications to connect to the new primary's actual IP address
   • Bootstrapped a third node as an asynchronous replica
   • Added a VIP managed by hand by the infrastructure team for manual failover
   • Verified data integrity and found no corruption
   • Migrated to a new Pacemaker-managed database cluster in a maintenance window and decommissioned the old one

   Action items

   • Express the Backup VIP rule as an INF preference for a replica instead of a -INF preference against the primary
   • Replace the VIPs with proxies on the application servers that route traffic by the state of the cluster
   • Run harsher fault injection tests of the Postgres cluster and consider automation that injects failures continually
   • Consider adding arbitrary restrictions to game day exercises, such as automation that has failed and a problem that cannot be diagnosed, so engineers practise bringing the service back another way

   Lessons

   A rule that two resources must not run together does not behave like the opposite of a rule that they must run together, and in some failure conditions the difference stopped Pacemaker from promoting a replica. A misconfiguration that shows only when two processes crash at almost the same time will not appear in basic tests or daily operations, only through fault injection. After two years in which the automation had handled every failure, the team's skill at controlling the cluster by hand had faded, and a runbook unused for years would likely have been out of date.

   View case
5. IL-0278 A weekly update put a Redis Cloud management process into an infinite loop, and most nodes of a GCP cluster failed

   Redis2016

   Symptoms

   On 8 February 2016 at 17:45 UTC, four nodes of the Redis Cloud cluster us-central1-1-1.gce.garantiadata.com on Google Cloud Platform stopped responding, four minutes after a first node had failed. The databases whose endpoints end in us-central1-1-1.gce.garantiadata.com went down.

   Service outageFailover failure

   Trigger

   The weekly upgrade of Redis Cloud's non-critical cluster components reached its first Google Cloud Platform cluster, us-central1-1-1.gce.garantiadata.com, at 17:30 UTC.

   Root cause

   Faulty deployment

   Component

   In-memory cacheDatabase failover managerAuthoritative DNS servers and records

   Contributing factor

   Overlapping failures

   Remediation

   • Shut down and terminated all nodes of the cluster
   • Deleted all of the cluster's endpoints from its DNS zone, removing by hand the entries left by throttled deletes
   • Provisioned new cluster nodes from scratch without the weekly update
   • Reconnected the original nodes' storage to the new nodes so persistence-enabled databases could be recovered
   • Bootstrapped and launched the databases, recovering the ones with persistence from storage

   Action items

   • Suspend all automatic weekly upgrades until a procedure that prevents this kind of outage is in place
   • Revalidate the QA processes for DevOps code deployments
   • Improve the tooling so it handles throttling of DNS requests

   View case

By root cause4