Incident Library

IL-0288 CDN77 · 2026-03-09databaseobservability

ZooKeeperWait rose to thousands of seconds on three redundant ClickHouse instances, degrading a CDN77 data pipeline

Almost all tables in the affected ClickHouse database are time series partitioned by a time unit, and writes from thousands of client machines are batched through Buffer tables. Two engineers, working from different places, independently found the cause. One looked for client machines that could explain normal batch sizes next to a falling mean and found two machines sending batches so small and so fast that they nearly halved the average. The other saw in system.part_log a sharp rise in created parts while the total number of rows stayed roughly the same, with partitions carrying timestamps as old as four months. The two machines had been unplugged from the network for several months with no other maintenance, and their shipper processes kept running. Their only data was a local health check on a regular cadence, near zero in volume, so months of it fit in the shipper's queue; the shipper batches at log time rather than send time, so the queue held many tiny batches, and the cadence evenly covered hours and days, the two most common partitioning keys. The Buffer table did combine these batches with other traffic, but each flush then spanned a long timeline that MergeTree partitioning cut into far too many parts: a table partitioned by 5 minutes gets 288 replicated parts for each historical day. The parts were inserted, replicated and immediately scheduled for TTL deletion, and the default max_number_of_merges_with_ttl_in_pool of 2 kept them from being cleared quickly, so Keeper traffic multiplied and slowed replication, though not ingest.

Most tables eventually cleared their backlog of parts on their own. The one short table partitioned by 5 minutes did not, and after the incident CDN77 dropped and recreated it and broadened its partitioning from minutes to hours. CDN77 added rejection of out-of-range timestamps in the root MaterializedView tables, with a second materialized view recording what was rejected, and built a test with an insert whose timestamps span days or months. It decided against lowering max_partitions_per_insert_block, because that setting can refuse inserts entirely and lose data. The incident also showed that Keeper operations grow much faster than linearly with the number of parts.

Severity
–
Scope
One large ClickHouse data pipeline was degraded for over an hour, with ZooKeeper traffic multiplied and replication slowed on three redundant instances; no observable impact on internal users, customers or partners
Users affected
none observed
Region
–
Time to detect
15m
Time to resolve
–

Symptoms

At 13:30 UTC on 9 March 2026, CDN77's data team was alerted to a sudden rise in the ZooKeeperWait metric on the ClickHouse cluster behind one of its large data pipelines. Alerting for this cluster size is set at 10 sec for the low watermark and 20 sec for the high one, and the metric reached thousands of seconds. The same rise appeared on three symmetrically redundant ClickHouse instances in different datacenters, each with its own ZooKeeper instance, and the pipeline stayed degraded for over an hour, with no observable impact on internal users, customers or partners.

The first suspect was a heavy read query stalling writes, the usual cause when ZooKeeperWaitMicroseconds climbs, especially as no configuration had changed that day. A daily data-diff job had started almost exactly when the metric went up, but within minutes its queries were light again while ZooKeeperWait stayed high, and CPU, RAM and the number of executing queries on the affected instances all looked normal, so heavy reads were ruled out.

About twenty minutes in, an engineer saw that the mean insert query size in system.query_log was falling: the log shippers seemed to send a median of around 70,000 rows per second, while ClickHouse reported an average around 40,000. The team set this aside as a side effect of Keeper load, because an overloaded ClickHouse normally shows slower Buffer engine flushes and these were not slowing. Then TTL merges rose sharply, with merge reason TTLDeleteMerge everywhere in system.part_log, and the error log repeated "Not executing log entry because 2 merges with TTL already executing." The team asked whether ttl_only_drop_parts, enabled on most of its tables, had somehow been disabled. The classic "too many parts" error never fired, because the configured limits were not reached.

Two gaps made the investigation harder. system.zookeeper_log, which would have shown Keeper traffic directly, was disabled, as it is by default in ClickHouse, because it can grow very large. The table that monitors delivery delay recorded only machines at CDN layer zero because of a leftover filter, while the machines behind the problem sat on a higher layer, so the lag was never measured precisely.

Symptom Replication lagThroughput degradation

Root cause

Root cause Capacity exhaustion

Component DatabaseCoordination and key-value storeDatabase replicationTelemetry agent
Contributing factor Monitoring blind spotUnreviewed defaultsIgnored early signal
Trigger
From 13:15 UTC two client machines that had been cut off from the network for several months came back and their log shippers sent months of queued, very small batches of health check data with old timestamps, spanning a very large time range.

Detection

An alert on the ZooKeeperWait metric fired at 13:30 UTC, fifteen minutes after the problematic inserts began at 13:15. The cluster's alerting uses a low watermark of 10 sec and a high watermark of 20 sec.

Remediation

Action items

Lessons

Data structure matters even in a pipeline built around volume: incoherent timestamps can destabilise it as effectively as changes in cardinality. A metric that looks odd and cannot be explained at once should not be set aside under incident pressure; the falling mean insert size was what led to the cause. Means and medians each have their place, and here the mean, not the median, exposed the problem. Partitioning has to be chosen carefully: too few partitions give oversized parts, slow mutations and costly or ineffective TTL, while too many slow column scans, add final merging to aggregating queries and overload the Keeper.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Coordination and key-value store, Database replication among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.