Cases10 cases

1 to 5 of 10

1. IL-0379 InfluxDB Cloud queries in Azure West Europe slowed and timed out as storage pods fell behind and kept restarting

   InfluxData2026

   Symptoms

   From the evening of 27 September 2026 (UTC), InfluxDB Cloud customers in Azure West Europe saw slow queries and query timeouts, and a query queue depth alert fired at 22:58 UTC. Writes were accepted the whole time, but new data on some partitions could not be queried right away.

   High latencyIntermittent timeoutsReplication lagProbe failures

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase replicationLiveness and readiness probesPod scheduling and placement

   Contributing factor

   Monitoring blind spotAutomation amplification

   Detection

   A query queue depth alert fired at 22:58 UTC on 27 September and an incident was opened.

   Remediation

   • Doubled query-layer capacity
   • Restarted stuck storage pods one at a time
   • Raised the readiness probe timeout and rolled it out fleet-wide
   • Doubled recovery concurrency after a restart
   • Raised the cleanup timeout to 24 hours, which stopped the forced restarts
   • Raised each storage pod's resource request so each host held fewer pods, doubling page cache memory and disk throughput per pod

   Action items

   • Add warning alerts for page-fault rate, cache coverage, disk reads on storage volumes and query abandonment
   • List major page faults in the runbooks as a leading indicator of storage performance degradation, with the relief valves that can be tuned
   • Keep tracking down the storage-engine reference leak
   • Move the disk check out of the health path

   Lessons

   Because the storage process uses little memory and the page cache takes what is left, a page cache too small for the data being queried does not show up as memory pressure on the usual memory metrics. InfluxData now treats major page faults as a leading indicator of storage performance degradation and alerts on cache pressure before customers see it.

   View case
2. IL-0490 Records were missing from LootLocker's new MySQL database after a migration cutover and the service degraded for 3 hours

   LootLocker2026sev2

   Symptoms

   About an hour after LootLocker cut over from a managed database provider to its own MySQL deployment on 2026-04-28, data anomalies showed up in production. A small window of writes made shortly before the migration, spanning 2026-04-19 to 2026-04-20, had never reached the new database, and the service stayed degraded for about three hours after the cutover.

   Data loss

   Trigger

   On 2026-04-20 at about 16:20 BST, a routine credential rotation, made as part of an unrelated configuration fix, changed the credentials that the temporary replication process used for its database connection, and the process crashed.

   Root cause

   Certificate or credential expiry

   Component

   DatabaseDatabase replication

   Contributing factor

   Change with a delayed effectMonitoring blind spotUnverified assumptionNo safeguard on destructive change

   Detection

   Data anomalies were detected in production at about 11:30 BST, about an hour after the cutover.

   Remediation

   • Restored normal service behaviour first and deferred the root cause analysis
   • Requested a restore of the deleted source database from the previous provider
   • Replayed the affected records for impacted users from the restored source database

   Action items

   • Perform technical changes of comparable risk in a scheduled maintenance window instead of as live cutovers
   • Require a pre-cutover checklist that verifies replication or sync processes are live and caught up just before cutover
   • Require an explicit downstream impact review for credential rotations of service accounts used by long-running processes
   • Keep source systems until integrity in the target has been independently verified after cutover
   • Publish planned maintenance and migration notices in the in-product News feed as well as Discord, with clear impact statements and more advance notice

   Lessons

   The failure was a silent side effect of an unrelated change made eight days earlier: the replication process recovered and kept working, so there was no signal that data had been dropped, and a pre-cutover integrity check would have caught it. Deleting the source database before validating the new one made recovery much harder. Choosing a live migration to avoid downtime left no clean point to verify integrity, and the resulting degradation and two days of piecemeal recovery far exceeded what a short planned maintenance window would have cost. The announcement on Discord was not clear enough about the impact and came with too little notice.

   View case
3. IL-0288 ZooKeeperWait rose to thousands of seconds on three redundant ClickHouse instances, degrading a CDN77 data pipeline

   CDN772026

   Symptoms

   At 13:30 UTC on 9 March 2026, CDN77's data team was alerted to a sudden rise in the ZooKeeperWait metric on the ClickHouse cluster behind one of its large data pipelines. Alerting for this cluster size is set at 10 sec for the low watermark and 20 sec for the high one, and the metric reached thousands of seconds. The same rise appeared on three symmetrically redundant ClickHouse instances in different datacenters, each with its own ZooKeeper instance, and the pipeline stayed degraded for over an hour, with no observable impact on internal users, customers or partners.

   Replication lagThroughput degradation

   Trigger

   From 13:15 UTC two client machines that had been cut off from the network for several months came back and their log shippers sent months of queued, very small batches of health check data with old timestamps, spanning a very large time range.

   Root cause

   Capacity exhaustion

   Component

   DatabaseCoordination and key-value storeDatabase replicationTelemetry agent

   Contributing factor

   Monitoring blind spotUnreviewed defaultsIgnored early signal

   Detection

   An alert on the ZooKeeperWait metric fired at 13:30 UTC, fifteen minutes after the problematic inserts began at 13:15. The cluster's alerting uses a low watermark of 10 sec and a high watermark of 20 sec.

   Remediation

   • Dropped and recreated the one table partitioned by 5 minutes that had not cleared its backlog of parts
   • Broadened that table's partitioning from minutes to hours

   Action items

   • Reject out-of-range timestamps in the root MaterializedView tables
   • Record statistics on rejected rows in a second materialized view
   • Refine the deliberately wide rejection boundaries as data on rejections comes in
   • Keep a minimal test scenario with an insert query whose timestamps span days or months
   • Make a range of observability improvements

   Lessons

   Data structure matters even in a pipeline built around volume: incoherent timestamps can destabilise it as effectively as changes in cardinality. A metric that looks odd and cannot be explained at once should not be set aside under incident pressure; the falling mean insert size was what led to the cause. Means and medians each have their place, and here the mean, not the median, exposed the problem. Partitioning has to be chosen carefully: too few partitions give oversized parts, slow mutations and costly or ineffective TTL, while too many slow column scans, add final merging to aggregating queries and overload the Keeper.

   View case
4. IL-0281 PostHog's Logs ClickHouse cluster silently deleted almost all its S3 data, losing US logs older than 3 days

   PostHog2026

   Symptoms

   On 19 February 2026, a node of the ClickHouse cluster behind PostHog's Logs product crashed, restarted and could not start again because it could not find a large amount of its data. Engineers then found that almost all of the Logs data stored on S3 was gone, so logs collected in the US region more than 3 days earlier were lost. Only the Logs product was affected.

   Data loss

   Trigger

   At 10:54 UTC a routine mutation was run to materialise an index. When it finished at 11:02, it set off a bug in ClickHouse's zero-copy replication.

   Root cause

   Software defect

   Component

   DatabaseDatabase replicationObject storageDatabase backups and snapshots

   Contributing factor

   Monitoring blind spotSingle point of failureUntested backup

   Detection

   No automated alarm fired. At 19:40 UTC a node in the cluster crashed, restarted and failed to start because of the missing data, and the engineers who investigated found that most of the S3-backed data was gone.

   Remediation

   • Switched Logs to a new table without zero-copy replication and caught up on recent messages
   • Backfilled the Logs data from the 3 days of Kafka history

   Action items

   • Disable zero-copy replication on all clusters
   • Audit the whole Logs ClickHouse configuration and keep experimental features out of production
   • Enable S3 versioning on the storage buckets so objects deleted by a database bug can be recovered
   • Spot-check configurations and data integrity strategies for single points of failure before a product becomes generally available

   Lessons

   Running Logs in its own cluster kept every other product unaffected, and 3 days of Kafka retention saved the recent data. An experimental setting chosen for an early internal stage was allowed to stay in production. Monitoring covered only incoming and recent data, so petabytes were deleted over 8 hours without an alarm. Relying on database replication alone over shared storage was a single point of failure, and S3 versioning would have let the deleted files be recovered.

   View case
5. IL-0392 MariaDB replication of section x1 between eqiad and codfw broke on a missing heartbeat row during a cookbook run

   Wikimedia Foundation2024

   Symptoms

   On 18 September 2024, while a cookbook was reconfiguring Wikimedia's core MariaDB databases from one-way eqiad -> codfw replication to circular replication ahead of a datacenter switchover, replication in the codfw -> eqiad direction broke on the x1 and es6 sections. Engineers watching the run noticed at 21:28 UTC that the x1 primary was not replicating.

   Replication lag

   Trigger

   Running the cookbook sre.switchdc.databases.prepare, which set up codfw -> eqiad replication and started pt-heartbeat, on the row-based x1 and es sections.

   Root cause

   Configuration change

   Component

   DatabaseDatabase replication

   Contributing factor

   Monitoring blind spotGap in automated testsHarmful manual action

   Detection

   Engineers watching the first run of the automation saw on orchestrator that the x1 primary was not replicating, because monitoring of circular replication was not yet enabled. The break in the eqiad -> codfw direction was reported by Icinga.

   Remediation

   • Skipped replication of the heartbeat tables with Replication_wild_ignore_table='heartbeat.%'
   • Inserted the missing heartbeat row with binary logging off on the es hosts
   • Inserted the missing heartbeat row with binary logging off on the x1 codfw hosts and removed the ignore rule

   Action items

   • Avoid the same issue again, probably by inserting the missing row when it is detected
   • Add a post-maintenance health check that replication works in both directions
   • Add options to run the maintenance in another order or on each section individually
   • Update the Phabricator task for each section rather than only at the end
   • Fix the monitoring gaps for circular replication
   • Made pt-heartbeat-wikimedia use STATEMENT format for its session on the x1 and row-based read-write es hosts, fixed in a later deploy

   Lessons

   Circular replication is not a normal working mode, so the first fix only moved the break to the other side; TRUNCATE would have worked better than DELETE because it always replicates as a statement, but it was not considered because it is more dangerous. Running sql_skip_slave_counter was avoided, since it could have made things worse or lost data. The check of the replication channel should also run once all tasks are complete, not only right after setup. The combination of row-based replication, pt-heartbeat and differing heartbeat rows was not described in earlier maintenance or documentation.

   View case

By root cause6