IL-0457 LINE · 2019-02-15 (published)database

The HA HDFS NameNode of a log-collection Hadoop cluster failed over around 09:00 after its ZKFC health check timed out

The Hadoop cluster, built on HDP-2.6.1.0 (2.6.1.0-129) with HDFS 2.7.3, collects the access and application logs of servers with fluentd and loads them into Apache Hive tables with an hourly ETL batch. fluentd writes log files into an HDFS directory, Hive LOAD DATA moves them into an intermediate table directory, Hive INSERT ... SELECT writes them into the final table in another file format, and Hive ALTER TABLE ... DROP PARTITION deletes the files of the intermediate table, which on this cluster moves them into the .Trash directory. With fs.trash.interval at 4320 minutes, the files under .Trash were deleted every 3 days, and that deletion held the NameNode's FSNamesystem write lock for long periods. RPC calls piled up in the client RPC port queue, and because the ZKFC health check used the same port, it timed out after 60 seconds and caused a failover.

The team chose two fixes. It changed fs.trash.interval from 4320 to 1440 so the files under .Trash are deleted every day, which shortens the time the deletion holds the write lock. It also set up the Service RPC port, a port used only for the health checks of the ZKFC and the DataNodes (including block reports), with dfs.namenode.servicerpc-address.mycluster.nn1=mynamenode1:8040 and dfs.namenode.servicerpc-address.mycluster.nn2=mynamenode2:8040, so the NameNode can answer the ZKFC health check without delay even when mass file deletion fills the client RPC port queue.

Severity: –
Scope: The HA HDFS NameNode of the log-collection Hadoop cluster failed over now and then around 09:00
Users affected: –
Region: –
Time to detect: –
Time to resolve: –

Symptoms

The NameNode of an HA HDFS cluster run by LINE Data Labs occasionally failed over at a particular time of day, around 09:00. At the failover the ZooKeeper Failover Controller (ZKFC) logged that its NameNode health check had timed out after 60 seconds: WARN ha.HealthMonitor (HealthMonitor.java:doHealthChecks(211)) - Transport-level exception trying to monitor health of NameNode at xxxx/xxx.xxx.xxx.xxx:xxxx: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read.

The NameNode log just before the failover showed that, at 09:00:30, the deletion of files under the .Trash directory had started with TrashPolicyDefault#deleteCheckpoint for trashRoot: hdfs://xxxx/user/hive/.Trash. Deleting one directory then held the lock for about 36 seconds, logged as FSNamesystem write lock held for 36094 ms via ... FSNamesystem.delete. More entries followed in which deleting blocks caused GC pauses of 1 to 7 seconds that kept the write lock held, such as Detected pause in JVM or host machine (eg GC): pause of approximately 1487ms and FSNamesystem write lock held for 2038 ms via ... FSNamesystem.removeBlocks.

While the write lock was held, about 3,500 unprocessed RPC calls sat in the queue of the NameNode RPC Client Port. The ZKFC log showed that its health check requests went to the same port as that queue, so the health check waited in the queue until it timed out and the ZKFC failed the NameNode over.

Symptom Unnecessary failover Lock contention or deadlock Intermittent timeouts

Root cause

Root cause Capacity exhaustion

Component Shared network filesystem Database failover manager

Contributing factor Unreviewed defaults

Trigger: Around 09:00 the NameNode began deleting the files under the .Trash checkpoint (TrashPolicyDefault#deleteCheckpoint), and deleting one directory held the FSNamesystem write lock for about 36 seconds.

Detection

–

Remediation

Changed fs.trash.interval from 4320 to 1440 so files under .Trash are deleted daily instead of every 3 days
Configured a dedicated Service RPC port for the ZKFC and DataNode health checks

Action items

–

Lessons

–

Sources

https://engineering.linecorp.com/ko/blog/data-engineering-software-troubleshooting · blog · archived copy

Related cases

IL-0137 Training initialisation on a 60-node NVIDIA B200 cluster took more than 8 hours instead of minutes because fragmented small random I/O from all nodes saturated the metadata service of the VAST Data NFS storageLablup Inc · 2026
same root cause · shared Shared network filesystem
IL-0129 Throughput of the 384-GPU SmolLM3 pretraining run collapsed because shared WekaFS storage evicted dataset shards to S3, then kept dropping on local NVMe because the nanotron dataloader built an index that grew with the step countHugging Face · 2025
same root cause · shared Shared network filesystem
IL-0321 GCP snapshots on all Patroni nodes stalled GitLab.com's primary PostgreSQL disk, Patroni failed over and 5xx spikedGitLab · 2021
same root cause · shared Database failover manager
IL-0320 Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayedPostmark · 2011
same root cause · shared Database failover manager

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Shared network filesystem, Database failover manager among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0457
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0457/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.