Incident Library
Applied filters ComponentDatabase failover managerOrganisationLINEClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0457 The HA HDFS NameNode of a log-collection Hadoop cluster failed over around 09:00 after its ZKFC health check timed out
    LINE2019
    Symptoms

    The NameNode of an HA HDFS cluster run by LINE Data Labs occasionally failed over at a particular time of day, around 09:00. At the failover the ZooKeeper Failover Controller (ZKFC) logged that its NameNode health check had timed out after 60 seconds: WARN ha.HealthMonitor (HealthMonitor.java:doHealthChecks(211)) - Transport-level exception trying to monitor health of NameNode at xxxx/xxx.xxx.xxx.xxx:xxxx: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read.

    Unnecessary failoverLock contention or deadlockIntermittent timeouts
    Trigger

    Around 09:00 the NameNode began deleting the files under the .Trash checkpoint (TrashPolicyDefault#deleteCheckpoint), and deleting one directory held the FSNamesystem write lock for about 36 seconds.

    Root cause
    Capacity exhaustion
    Component
    Shared network filesystemDatabase failover manager
    Contributing factor
    Unreviewed defaults
    Remediation
    • Changed fs.trash.interval from 4320 to 1440 so files under .Trash are deleted daily instead of every 3 days
    • Configured a dedicated Service RPC port for the ZKFC and DataNode health checks
    View case
Check your cluster for this

This case involves Database failover manager and Shared network filesystem. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
Database failover managerShared network filesystem
Contributing factor
Unreviewed defaults
Organisation
LINE
  1. IL-0457 The HA HDFS NameNode of a log-collection Hadoop cluster failed over around 09:00 after its ZKFC health check timed out
    LINE2019