Cases18 cases

3 to 7 of 18

1. IL-0306 A Neon pageserver's EC2 instance failed without warning and Postgres databases of 0.4% of us-east-1 projects stalled

   Neon2024

   Symptoms

   At 17:28 on 2024-08-14 the EC2 instance pageserver-7.us-east-1.aws.neon.tech stopped responding. Neon customer databases attached to that pageserver stalled whenever they needed to read data that was not already in their Postgres buffer cache.

   Storage disconnected or stale mountsHigh error rate

   Trigger

   The EC2 instance hosting pageserver-7 in us-east-1 failed without the advance warning AWS usually gives before a hardware failure.

   Root cause

   Hardware failure

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Partial rollout

   Detection

   Several alerts fired at 17:38, in Slack and as automatic pages to the on-call engineers: the pageserver stopped answering metric scrapes, the control plane API error rate rose, and user operations such as starting endpoints and creating timelines got stuck.

   Remediation

   • Migrated the 17037 projects on pageserver-7 to other pageservers in the region with a semi-manual script
   • Brought back by hand about 20 projects whose in-flight operations blocked the migration

   Action items

   • Move all paying customers onto the storage controller sooner than planned
   • Finish moving the existing pageserver infrastructure under the storage controller

   Lessons

   Neon's first control plane design, in which operations always run to completion, cannot drive a fast response to node failures at scale, because operations that depend on physical machines may never complete. A human decision in the loop and alerts with several minutes of delay made the response too slow; the storage controller, with its own heartbeats and automatic rescheduling, moves work off a failed pageserver in seconds.

   View case
2. IL-0367 Azure Storage in Central US rejected VM disk I/O, taking down VMs and the SQL Database and Cosmos DB services on them

   Microsoft Azure2024

   Symptoms

   From 21:40 UTC on 18 July 2024, virtual machines in the Azure Central US region lost access to their disks, and the services running on those VMs failed. Azure SQL Database and SQL Managed Instance showed unhealthy nodes and login failures, Cosmos DB requests failed, and service management operations failed across many Azure services.

   Storage disconnected or stale mountsService outageHigh error rateFailover failureStale endpoint routing

   Trigger

   Routine changes to the VM host fleet led to a new allow list of VM host address ranges being generated and published to all Storage scale units in Central US. Recent network configuration changes had made some of the infrastructure file servers that hold the source data unreachable from the server generating the list, so the published list was incomplete.

   Root cause

   Configuration change

   Component

   Cloud block storageDatabaseDatabase failover manager

   Contributing factor

   Unsafe failure behaviourNo staged rolloutExcessive blast radiusMonitoring blind spot

   Detection

   Service monitoring detected the drop in VM availability at 22:06 UTC while the allow list update was being published. SQL monitoring detected unhealthy nodes and login failures within one minute of SQL becoming unavailable, and monitoring flagged the availability drop for Cosmos DB, SQL DB and SQL DB Managed Instance at 22:14 UTC.

   Remediation

   • Halted the allow list update workflow worldwide
   • Paused all deployments in the Central US region
   • Paused compute buildout deployments in all regions
   • Updated the configuration of all Storage scale units in Central US, restoring storage availability by 02:55 UTC
   • Started geo-failovers for SQL databases with automatic failover policies
   • Restarted SQL gateway nodes to refresh their caches and route connections to the right nodes
   • Forced completion of SQL failovers that had not completed
   • Failed over the Cosmos DB control plane and active-passive accounts with a write region in Central US
   • Cancelled customer-triggered graceful switch region operations that blocked Cosmos DB failovers
   • Removed locks on Cosmos DB service metadata to force-complete stuck failovers
   • Repaired impacted Cosmos DB partitions for single-region accounts
   • Repaired backend nodes and failed Cosmos DB accounts back to Central US, redoing failback where it failed

   Action items

   • Make the allow list generation workflow detect incomplete source information and halt
   • Alert on storage requests rejected by allow list checks
   • Deploy allow lists one Availability Zone and storage type at a time, over 24 hours
   • Add VM health checks and an automatic stop to the allow list deployment workflow
   • Reevaluate the policy for Microsoft-managed failover of SQL failover groups
   • Fix the Cosmos DB failback workflow that broke some MongoDB drivers for MongoDB API accounts
   • Fix Cosmos DB failback for accounts with private endpoints so it does not need redoing
   • Make the storage data-plane firewall reject invalid allow lists and keep the last-known-good state
   • Improve Azure NetApp Files monitors so impacting events are detected and classified in time
   • Monitor more Azure NetApp Files service metrics to correlate similar events faster
   • Make the SQL Service Fabric cluster location change notifications reliable under load
   • Make SQL geo-failover workflows more robust so they complete
   • Remove the issues that delayed 5% of Cosmos DB failovers
   • Migrate Azure DevOps customer metadata to the right geography
   • Decouple Azure NetApp Files regional reads and writes to keep this class of issue to one region
   • Evaluate caching to reduce Azure NetApp Files reliance on stored utilization data
   • Add automatic per-partition failover for Cosmos DB multi-region active-passive accounts
   • Build the Resilient Ephemeral OS disk improvement so VMs survive storage incidents better

   View case
3. IL-0291 GitLab.com's redis-sidekiq Sentinel could not elect a primary after a GCP host error, so Sidekiq was down for 30 minutes

   GitLab2023

   Symptoms

   GitLab.com showed widespread, erratic behaviour and more 500 errors for about 30 minutes while its redis-sidekiq Redis was down. Alerts in the #production channel began with Redis client errors and spread to error SLO violations on several services, such as the Webservice and API load balancer and Puma.

   Failover failureHigh error rate

   Trigger

   The VM of the primary node, redis-sidekiq-03-db-gprd, failed on a GCP host error and was reset and moved to another node.

   Root cause

   Cloud provider outage

   Component

   In-memory cacheDatabase failover manager

   Contributing factor

   Single point of failureMonitoring blind spot

   Detection

   Several alerts in the #production channel, first Redis client errors and then error SLO violations on several services, such as the Webservice and API load balancer and Puma.

   Remediation

   • Restarted the failed Redis node so Sentinel had enough nodes to elect a primary

   Action items

   • Alert when the Sentinel service is down
   • Make merge requests robust to Redis Sidekiq outages

   Lessons

   An alert on a down Sentinel node would have kept enough nodes up to form a quorum when failover was needed, and could have avoided the incident entirely. Removing a noisy alert earlier may have let the down node go unnoticed.

   View case
4. IL-0295 Pusher's Channels publish API on the mt1 cluster was largely unavailable for 11 minutes while a Redis master failed over

   Pusher2022

   Symptoms

   At 10:42 UTC on 11 April 2022, the API for publishing messages to Pusher Channels on the mt1 cluster became largely unavailable, and from about 10:44 requests to the MT1 API returned more errors.

   Service outageHigh latencySplit brain

   Trigger

   An engineer restarted the master node of an mt1 Redis group to apply a larger replication buffer, after restarting both replicas of that group.

   Root cause

   Software defect

   Component

   In-memory cacheDatabase failover manager

   Contributing factor

   Unverified assumptionUnrepresentative staging

   Detection

   Pusher saw error rates rise for requests to the MT1 API from about 10:44 UTC.

   Action items

   • Fix the Redis shutdown script to check that failover has completed and the node is no longer a master before terminating

   View case
5. IL-0321 GCP snapshots on all Patroni nodes stalled GitLab.com's primary PostgreSQL disk, Patroni failed over and 5xx spiked

   GitLab2021

   Symptoms

   At 23:38 UTC on 2021-01-16 GitLab.com error rates spiked, and a minute later multiple alerts for service disruption arrived, two minutes after GCP disk snapshots had been started on all Patroni database nodes. Some requests that needed database writes returned 5xx errors for about four to five minutes, while page refreshes checked during the incident responded normally.

   High error rate

   Trigger

   During the change production#2648, at 23:36 UTC an engineer started GCP disk snapshots on all Patroni nodes at the same time, as a safety step before a change to the filesystem, believing the snapshot was a transparent background operation through the GCP API.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase failover managerCloud block storage

   Contributing factor

   Unverified assumptionNo staged rollout

   Detection

   A flood of alerts about downstream impact; multiple service disruption alerts arrived at 23:39 UTC, a minute after the error spikes began. The incident was declared in Slack at 23:56.

   Remediation

   • Patroni invalidated patroni-06's lock and promoted patroni-03 as the new primary
   • Cancelled the snapshot requests in two of the three zones

   Action items

   • Update the Patroni and PostgreSQL runbooks to clarify GCP snapshot best practices
   • Add a checklist for change management reviewers and approvers to the runbooks, referenced in the handbook
   • Schedule a call with Google about IO sensitivities

   Lessons

   Do not take a GCP snapshot of a running PostgreSQL leader. Technical reviewers have to review every change or addition to a change request's description before a manager or director approves it. The Datastores team should explain in detail how backups and snapshots work on the production database. No action belonging to a change should start before the change's scheduled start.

   View case

By root cause9