Cases6 cases

3 to 6 of 6

1. IL-0291 GitLab.com's redis-sidekiq Sentinel could not elect a primary after a GCP host error, so Sidekiq was down for 30 minutes

   GitLab2023

   Symptoms

   GitLab.com showed widespread, erratic behaviour and more 500 errors for about 30 minutes while its redis-sidekiq Redis was down. Alerts in the #production channel began with Redis client errors and spread to error SLO violations on several services, such as the Webservice and API load balancer and Puma.

   Failover failureHigh error rate

   Trigger

   The VM of the primary node, redis-sidekiq-03-db-gprd, failed on a GCP host error and was reset and moved to another node.

   Root cause

   Cloud provider outage

   Component

   In-memory cacheDatabase failover manager

   Contributing factor

   Single point of failureMonitoring blind spot

   Detection

   Several alerts in the #production channel, first Redis client errors and then error SLO violations on several services, such as the Webservice and API load balancer and Puma.

   Remediation

   • Restarted the failed Redis node so Sentinel had enough nodes to elect a primary

   Action items

   • Alert when the Sentinel service is down
   • Make merge requests robust to Redis Sidekiq outages

   Lessons

   An alert on a down Sentinel node would have kept enough nodes up to form a quorum when failover was needed, and could have avoided the incident entirely. Removing a noisy alert earlier may have let the down node go unnoticed.

   View case
2. IL-0321 GCP snapshots on all Patroni nodes stalled GitLab.com's primary PostgreSQL disk, Patroni failed over and 5xx spiked

   GitLab2021

   Symptoms

   At 23:38 UTC on 2021-01-16 GitLab.com error rates spiked, and a minute later multiple alerts for service disruption arrived, two minutes after GCP disk snapshots had been started on all Patroni database nodes. Some requests that needed database writes returned 5xx errors for about four to five minutes, while page refreshes checked during the incident responded normally.

   High error rate

   Trigger

   During the change production#2648, at 23:36 UTC an engineer started GCP disk snapshots on all Patroni nodes at the same time, as a safety step before a change to the filesystem, believing the snapshot was a transparent background operation through the GCP API.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase failover managerCloud block storage

   Contributing factor

   Unverified assumptionNo staged rollout

   Detection

   A flood of alerts about downstream impact; multiple service disruption alerts arrived at 23:39 UTC, a minute after the error spikes began. The incident was declared in Slack at 23:56.

   Remediation

   • Patroni invalidated patroni-06's lock and promoted patroni-03 as the new primary
   • Cancelled the snapshot requests in two of the three zones

   Action items

   • Update the Patroni and PostgreSQL runbooks to clarify GCP snapshot best practices
   • Add a checklist for change management reviewers and approvers to the runbooks, referenced in the handbook
   • Schedule a call with Google about IO sensitivities

   Lessons

   Do not take a GCP snapshot of a running PostgreSQL leader. Technical reviewers have to review every change or addition to a change request's description before a manager or director approves it. The Datastores team should explain in detail how backups and snapshots work on the production database. No action belonging to a change should start before the change's scheduled start.

   View case
3. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicas

   Cloudflare2020

   Symptoms

   On 2 November 2020 the Cloudflare API and dashboard were degraded for six hours and 33 minutes. The success rate of API queries dropped at times to 75%, and the dashboard was up to 80 times slower than normal, while Cloudflare's edge network kept working.

   High error rateHigh latencyDatabase overloadWrites rejectedPacket loss

   Trigger

   At 14:43 UTC a network switch in a control plane data center partly failed: it kept LACP and BGP running but stopped forwarding some packets, and it recovered on its own six minutes later.

   Root cause

   DNS or network fault

   Component

   NIC, cable and switch port linkCoordination and key-value storeDatabase failover managerDatabaseDatabase replication

   Contributing factor

   Automation amplificationNo headroomHarmful manual action

   Detection

   At 14:43 UTC alerts fired because the misbehaving switch was unreachable to pings. Engineers then saw that the primary authentication database was overwhelmed.

   Remediation

   • Shed load from the primary authentication database by slowing SSL certificate pushes to the edge, emails and other features
   • Moved API traffic that could use read replicas to the secondary data center by hand
   • Sent authentication calls fully back to the primary data center when the dashboard got worse
   • Rebuilt the first database replica, which put itself back into service at 21:20 UTC and returned performance to normal
   • Ramped the turned-down services back up so asynchronous processing could catch up

   Action items

   • Remove the cluster management limitation that requires rebuilding all replicas after a primary is promoted
   • Redesign the user sessions system so dashboard traffic can be steered between data centers
   • Change the auto-remediation configuration so a database replica is not promoted so quickly
   • Tune each system involved to be more resilient to failures of its dependencies

   Lessons

   Every system in the chain had redundancy and none failed completely; each only degraded, which made the chain of events much harder to model and foresee. Automating replica promotion so that it could trigger on a minute's notice had been a point of pride, but for at least one database a fast promotion can do more harm than the failure it answers. Most general-purpose cluster managers use Raft or Paxos rather than Byzantine fault tolerant protocols, accepting a rare failure mode in exchange for a protocol that is simpler to implement correctly and debug.

   View case
4. IL-0241 A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one day

   Stripe2019

   Symptoms

   On 10 July 2019 the Stripe API failed a substantial majority of requests twice, from 16:35 to 17:02 UTC and from 21:14 to 22:47 UTC. In the first period automated monitoring detected within a minute that a database cluster had failed to elect a new primary.

   High error rateFailover failureWrites rejectedIntermittent timeoutsSustained high node CPU

   Trigger

   The primary node of a database cluster failed at 16:35 UTC while one shard had several stalled nodes. The second period was set off by rolling the election protocol back to the previous minor version at 20:42 UTC.

   Root cause

   Software defect

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Change with a delayed effectMonitoring blind spotExcessive blast radiusHarmful manual actionUnverified assumptionOverlapping failures

   Detection

   Automated monitoring detected the failed election within a minute, and the team was alerted at 16:36 UTC. The second period was announced at 21:14 UTC by automated alerts that some shards in the cluster were unavailable.

   Remediation

   • Restarted all nodes of the database cluster, which forced a successful primary election
   • Updated the production configuration and restarted the database cluster's nodes
   • Ramped traffic back up, putting services for user-initiated API requests first

   Action items

   • Alert when nodes stop reporting their replication lag
   • Alert when a shard enters a state that could trigger the election fault
   • Work with the database maintainers on a fix for the underlying fault
   • Add circuit breaking on failed operations to particular database clusters
   • Use fault isolation to contain the impact of a single failed shard
   • Limit the resources used by clients that keep retrying failed requests
   • Add procedures and tooling that make rapid configuration changes during incident response safer

   View case

By root cause6