Cases2 cases

1 to 2 of 2

1. IL-0457 The HA HDFS NameNode of a log-collection Hadoop cluster failed over around 09:00 after its ZKFC health check timed out

   LINE2019

   Symptoms

   The NameNode of an HA HDFS cluster run by LINE Data Labs occasionally failed over at a particular time of day, around 09:00. At the failover the ZooKeeper Failover Controller (ZKFC) logged that its NameNode health check had timed out after 60 seconds: WARN ha.HealthMonitor (HealthMonitor.java:doHealthChecks(211)) - Transport-level exception trying to monitor health of NameNode at xxxx/xxx.xxx.xxx.xxx:xxxx: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read.

   Unnecessary failoverLock contention or deadlockIntermittent timeouts

   Trigger

   Around 09:00 the NameNode began deleting the files under the .Trash checkpoint (TrashPolicyDefault#deleteCheckpoint), and deleting one directory held the FSNamesystem write lock for about 36 seconds.

   Root cause

   Capacity exhaustion

   Component

   Shared network filesystemDatabase failover manager

   Contributing factor

   Unreviewed defaults

   Remediation

   • Changed fs.trash.interval from 4320 to 1440 so files under .Trash are deleted daily instead of every 3 days
   • Configured a dedicated Service RPC port for the ZKFC and DataNode health checks

   View case
2. IL-0431 During Mackerel's move to AWS, PostgreSQL failed over to a replica that had stopped replicating, losing an hour of data

   Hatena2017

   Symptoms

   On 7 August 2017, while checking the application after the maintenance that moved Mackerel from an on-premises data center to AWS, the team found that data updated during a specific time window was missing. Everything written to the PostgreSQL database between 14:30 and 15:30 JST was gone.

   Data lossReplication lagNetwork unreachable

   Trigger

   One hour after the application was switched to the newly promoted primary db001, db001 lost network connectivity and the database failed over to the standby db002.

   Root cause

   Operator error

   Component

   DatabaseDatabase replicationDatabase failover manager

   Contributing factor

   Overlapping failuresUnreviewed defaultsUnrepresentative stagingAutomation amplification

   Detection

   The team found the missing data while checking how the application behaved after the migration work.

   Remediation

   • Tried to restore the lost updates by hand from the table data left on db001 and gave up
   • Tried point-in-time recovery from the base backup and WAL and gave up
   • Built a new replica db003 with db002 as primary to restore redundancy
   • Kept db001 in maintenance state for investigation
   • Watched db002 for some time to confirm the problem did not recur, then completed the migration

   Action items

   • Change the PostgreSQL replica creation script to set recovery_target_timeline = 'latest' in recovery.conf by default
   • Add a check to a server configuration test tool such as Serverspec that recovery.conf contains recovery_target_timeline = 'latest'

   Lessons

   A setting needed only for a rare layout, such as a grandchild replica during a migration, was applied by hand through a script option and was forgotten. Because recovery_target_timeline = 'latest' is harmless on ordinary standbys too, Hatena now sets it by default whenever a replica is created. The staging rehearsal used the same layout and steps but had the setting right, so it could not catch the mistake. Hatena notes that a similar layout may be used again for a future PostgreSQL upgrade or a move to another region.

   View case

By root cause2