Cases3 cases

1 to 3 of 3

1. IL-0321 GCP snapshots on all Patroni nodes stalled GitLab.com's primary PostgreSQL disk, Patroni failed over and 5xx spiked

   GitLab2021

   Symptoms

   At 23:38 UTC on 2021-01-16 GitLab.com error rates spiked, and a minute later multiple alerts for service disruption arrived, two minutes after GCP disk snapshots had been started on all Patroni database nodes. Some requests that needed database writes returned 5xx errors for about four to five minutes, while page refreshes checked during the incident responded normally.

   High error rate

   Trigger

   During the change production#2648, at 23:36 UTC an engineer started GCP disk snapshots on all Patroni nodes at the same time, as a safety step before a change to the filesystem, believing the snapshot was a transparent background operation through the GCP API.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase failover managerCloud block storage

   Contributing factor

   Unverified assumptionNo staged rollout

   Detection

   A flood of alerts about downstream impact; multiple service disruption alerts arrived at 23:39 UTC, a minute after the error spikes began. The incident was declared in Slack at 23:56.

   Remediation

   • Patroni invalidated patroni-06's lock and promoted patroni-03 as the new primary
   • Cancelled the snapshot requests in two of the three zones

   Action items

   • Update the Patroni and PostgreSQL runbooks to clarify GCP snapshot best practices
   • Add a checklist for change management reviewers and approvers to the runbooks, referenced in the handbook
   • Schedule a call with Google about IO sensitivities

   Lessons

   Do not take a GCP snapshot of a running PostgreSQL leader. Technical reviewers have to review every change or addition to a change request's description before a manager or director approves it. The Datastores team should explain in detail how backups and snapshots work on the production database. No action belonging to a change should start before the change's scheduled start.

   View case
2. IL-0457 The HA HDFS NameNode of a log-collection Hadoop cluster failed over around 09:00 after its ZKFC health check timed out

   LINE2019

   Symptoms

   The NameNode of an HA HDFS cluster run by LINE Data Labs occasionally failed over at a particular time of day, around 09:00. At the failover the ZooKeeper Failover Controller (ZKFC) logged that its NameNode health check had timed out after 60 seconds: WARN ha.HealthMonitor (HealthMonitor.java:doHealthChecks(211)) - Transport-level exception trying to monitor health of NameNode at xxxx/xxx.xxx.xxx.xxx:xxxx: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read.

   Unnecessary failoverLock contention or deadlockIntermittent timeouts

   Trigger

   Around 09:00 the NameNode began deleting the files under the .Trash checkpoint (TrashPolicyDefault#deleteCheckpoint), and deleting one directory held the FSNamesystem write lock for about 36 seconds.

   Root cause

   Capacity exhaustion

   Component

   Shared network filesystemDatabase failover manager

   Contributing factor

   Unreviewed defaults

   Remediation

   • Changed fs.trash.interval from 4320 to 1440 so files under .Trash are deleted daily instead of every 3 days
   • Configured a dedicated Service RPC port for the ZKFC and DataNode health checks

   View case
3. IL-0320 Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayed

   Postmark2011

   Symptoms

   At about 8:15am EDT Postmark's primary MySQL server ran out of disk space, and email sending, the API and the web interface were delayed. Disk space alerts fired, but the space had grown much faster than usual and MySQL was already struggling when the alerts were noticed. MySQL kept running while it failed to write some data to disk, so MySQL-MMM, which normally detects problems and fails over to a healthy server, never triggered a failover; the team spent about 30 minutes trying to free logs and disk space before failing over by hand.

   Node disk fullWrites rejectedFailover failure

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase failover manager

   Contributing factor

   –

   Detection

   Per-server monitoring raised disk space alerts, but the disk filled so quickly that MySQL was already struggling by the time the alerts were noticed.

   Remediation

   • Tried to free up logs and disk space on the primary server
   • Failed over manually to the second MySQL server
   • Recovered the jobs that had failed
   • Sent the emails that each server had queued locally while the database was unavailable

   Action items

   • Upgrade the MySQL cluster with a focus on availability and performance
   • Make the alerts more sensitive to catch problems sooner

   Lessons

   Looking back, Postmark said it should have failed over to the second server right away instead of first making sure replication was healthy.

   View case

By root cause1