Cases2 cases

1 to 2 of 2

1. IL-0288 ZooKeeperWait rose to thousands of seconds on three redundant ClickHouse instances, degrading a CDN77 data pipeline

   CDN772026

   Symptoms

   At 13:30 UTC on 9 March 2026, CDN77's data team was alerted to a sudden rise in the ZooKeeperWait metric on the ClickHouse cluster behind one of its large data pipelines. Alerting for this cluster size is set at 10 sec for the low watermark and 20 sec for the high one, and the metric reached thousands of seconds. The same rise appeared on three symmetrically redundant ClickHouse instances in different datacenters, each with its own ZooKeeper instance, and the pipeline stayed degraded for over an hour, with no observable impact on internal users, customers or partners.

   Replication lagThroughput degradation

   Trigger

   From 13:15 UTC two client machines that had been cut off from the network for several months came back and their log shippers sent months of queued, very small batches of health check data with old timestamps, spanning a very large time range.

   Root cause

   Capacity exhaustion

   Component

   DatabaseCoordination and key-value storeDatabase replicationTelemetry agent

   Contributing factor

   Monitoring blind spotUnreviewed defaultsIgnored early signal

   Detection

   An alert on the ZooKeeperWait metric fired at 13:30 UTC, fifteen minutes after the problematic inserts began at 13:15. The cluster's alerting uses a low watermark of 10 sec and a high watermark of 20 sec.

   Remediation

   • Dropped and recreated the one table partitioned by 5 minutes that had not cleared its backlog of parts
   • Broadened that table's partitioning from minutes to hours

   Action items

   • Reject out-of-range timestamps in the root MaterializedView tables
   • Record statistics on rejected rows in a second materialized view
   • Refine the deliberately wide rejection boundaries as data on rejections comes in
   • Keep a minimal test scenario with an insert query whose timestamps span days or months
   • Make a range of observability improvements

   Lessons

   Data structure matters even in a pipeline built around volume: incoherent timestamps can destabilise it as effectively as changes in cardinality. A metric that looks odd and cannot be explained at once should not be set aside under incident pressure; the falling mean insert size was what led to the cause. Means and medians each have their place, and here the mean, not the median, exposed the problem. Partitioning has to be chosen carefully: too few partitions give oversized parts, slow mutations and costly or ineffective TTL, while too many slow column scans, add final merging to aggregating queries and overload the Keeper.

   View case
2. IL-0443 A Repro Aurora MySQL reader restarted unexpectedly and writer insert latency rose about sevenfold for a day or more

   Repro2019

   Symptoms

   A reader of Repro's Aurora MySQL cluster restarted for no known reason, and right after that the writer's insert latency rose to about seven times normal and writer performance stayed badly degraded for a long time. This happened more than once, twice within two weeks by 11 November 2019. While the writer was slow, the applications that process data collected by the Repro SDK fell behind, and the team could only watch until the writer recovered.

   High latencyReplication lagThroughput degradation

   Trigger

   Long-running queries on the reader; the author recalls that the cause was a query sent to the reader from Redash.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase replication

   Contributing factor

   Monitoring blind spot

   Remediation

   • Waited for writer performance to return, since the rollback segment release could not be sped up

   Action items

   • Add a monitor on the rollback segment, mysql.innodb.history_list_length in Datadog
   • Kill queries that run long enough to make the reader restart as soon as they are found
   • Set max_execution_time so queries are stopped after a fixed time

   Lessons

   Once the rollback segment has grown large, nothing speeds up its release across the cluster, so the way to avoid the latency is to keep transactions small and stop long-running queries before the rollback segment grows.

   View case

By root cause1