Cases4 cases

1 to 4 of 4

1. IL-0379 InfluxDB Cloud queries in Azure West Europe slowed and timed out as storage pods fell behind and kept restarting

   InfluxData2026

   Symptoms

   From the evening of 27 September 2026 (UTC), InfluxDB Cloud customers in Azure West Europe saw slow queries and query timeouts, and a query queue depth alert fired at 22:58 UTC. Writes were accepted the whole time, but new data on some partitions could not be queried right away.

   High latencyIntermittent timeoutsReplication lagProbe failures

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase replicationLiveness and readiness probesPod scheduling and placement

   Contributing factor

   Monitoring blind spotAutomation amplification

   Detection

   A query queue depth alert fired at 22:58 UTC on 27 September and an incident was opened.

   Remediation

   • Doubled query-layer capacity
   • Restarted stuck storage pods one at a time
   • Raised the readiness probe timeout and rolled it out fleet-wide
   • Doubled recovery concurrency after a restart
   • Raised the cleanup timeout to 24 hours, which stopped the forced restarts
   • Raised each storage pod's resource request so each host held fewer pods, doubling page cache memory and disk throughput per pod

   Action items

   • Add warning alerts for page-fault rate, cache coverage, disk reads on storage volumes and query abandonment
   • List major page faults in the runbooks as a leading indicator of storage performance degradation, with the relief valves that can be tuned
   • Keep tracking down the storage-engine reference leak
   • Move the disk check out of the health path

   Lessons

   Because the storage process uses little memory and the page cache takes what is left, a page cache too small for the data being queried does not show up as memory pressure on the usual memory metrics. InfluxData now treats major page faults as a leading indicator of storage performance degradation and alerts on cache pressure before customers see it.

   View case
2. IL-0305 Replicas of one Intercom Vitess shard hit 100% CPU and fell up to 2.2 hours behind, serving stale inbox data

   Intercom2026

   Symptoms

   From 17:03 UTC on January 27, 2026, a small subset of Intercom customers, fewer than 1%, saw inconsistent latency and stale data when reading from the platform. At 17:25 UTC an alarm on an excessive message backlog in a worker declared an incident automatically.

   Replication lagDatabase overloadHigh latency

   Trigger

   An internal app deletion process, running more concurrent deletion steps per app than usual with no rate limiting, and the ETL data pipeline both put heavy load on the same database shard at the same time.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase replication

   Contributing factor

   No headroom

   Detection

   An alarm on an excessive message backlog in a worker fired at 17:25 UTC and declared an incident automatically.

   Remediation

   • Disabled ETL processing to reduce load on the affected shard
   • Scaled the app deletion workers to zero
   • Re-enabled ETL carefully once monitoring showed stable metrics

   Action items

   • Increase the capacity of shards to leave headroom for unexpected load spikes
   • Add rate limiting and resource controls to the app deletion process so it backs off when database load is high
   • Review how background work such as ETL and maintenance tasks competes with customer-facing operations for database capacity
   • Alert on replication lag earlier and link it to specific workloads
   • Communicate proactively with affected customers in incidents with a limited blast radius

   Lessons

   Because data is spread across many independent shards, the overload stayed on one shard and reached fewer than 1% of customers instead of the whole platform.

   View case
3. IL-0443 A Repro Aurora MySQL reader restarted unexpectedly and writer insert latency rose about sevenfold for a day or more

   Repro2019

   Symptoms

   A reader of Repro's Aurora MySQL cluster restarted for no known reason, and right after that the writer's insert latency rose to about seven times normal and writer performance stayed badly degraded for a long time. This happened more than once, twice within two weeks by 11 November 2019. While the writer was slow, the applications that process data collected by the Repro SDK fell behind, and the team could only watch until the writer recovered.

   High latencyReplication lagThroughput degradation

   Trigger

   Long-running queries on the reader; the author recalls that the cause was a query sent to the reader from Redash.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase replication

   Contributing factor

   Monitoring blind spot

   Remediation

   • Waited for writer performance to return, since the rollback segment release could not be sped up

   Action items

   • Add a monitor on the rollback segment, mysql.innodb.history_list_length in Datadog
   • Kill queries that run long enough to make the reader restart as soon as they are found
   • Set max_execution_time so queries are stopped after a fixed time

   Lessons

   Once the rollback segment has grown large, nothing speeds up its release across the cluster, so the way to avoid the latency is to keep transactions small and stop long-running queries before the rollback segment grows.

   View case
4. IL-0296 Transloadit's us-east Redis cluster lost queued jobs and heartbeats, raising WORKER_JOB_ERROR errors for 81 customers

   Transloadit2016

   Symptoms

   Reports came in of more WORKER_JOB_ERROR errors, which tell Transloadit there is a communication problem between its Uploaders, the machines that receive files and orchestrate Assemblies, and its Drones, the machines that run encoding tasks. Many Assemblies became much slower because retries with exponential back-off kicked in, and a few could not be replayed.

   High error rateHigh latencyLoad imbalance

   Trigger

   A steep increase in HLS traffic, together with more Drones online processing the same volume faster, which put more load on the central Redis nodes.

   Root cause

   Capacity exhaustion

   Component

   In-memory cacheDatabase replication

   Contributing factor

   Monitoring blind spotUnverified assumptionHarmful manual action

   Detection

   Reports of an increased number of WORKER_JOB_ERROR errors.

   Remediation

   • Re-added lost messages and relaxed heartbeat monitoring as a stopgap
   • Upgraded Redis to the latest stable version
   • Doubled the Redis cluster to six nodes and then added three more, which made it worse
   • Brought the number of read replicas down to one, which cut traffic and failure rates
   • Capped how far Drones scale up in each region

   Action items

   • Add logging and alarms for approaching saturation
   • Find ways to decentralise the traffic between the Redis nodes and the Drones and Uploaders
   • Add sharding per Uploader on top of the sharding per region

   Lessons

   To handle more traffic Transloadit had to use fewer Redis nodes, not more: each added replica multiplied the replication traffic from the single leader that takes every write, so scaling up the number of nodes was the wrong call. Bandwidth had not been among the monitored Redis metrics.

   View case

By root cause1