Cases12 cases

1 to 5 of 12

1. IL-0379 InfluxDB Cloud queries in Azure West Europe slowed and timed out as storage pods fell behind and kept restarting

   InfluxData2026

   Symptoms

   From the evening of 27 September 2026 (UTC), InfluxDB Cloud customers in Azure West Europe saw slow queries and query timeouts, and a query queue depth alert fired at 22:58 UTC. Writes were accepted the whole time, but new data on some partitions could not be queried right away.

   High latencyIntermittent timeoutsReplication lagProbe failures

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase replicationLiveness and readiness probesPod scheduling and placement

   Contributing factor

   Monitoring blind spotAutomation amplification

   Detection

   A query queue depth alert fired at 22:58 UTC on 27 September and an incident was opened.

   Remediation

   • Doubled query-layer capacity
   • Restarted stuck storage pods one at a time
   • Raised the readiness probe timeout and rolled it out fleet-wide
   • Doubled recovery concurrency after a restart
   • Raised the cleanup timeout to 24 hours, which stopped the forced restarts
   • Raised each storage pod's resource request so each host held fewer pods, doubling page cache memory and disk throughput per pod

   Action items

   • Add warning alerts for page-fault rate, cache coverage, disk reads on storage volumes and query abandonment
   • List major page faults in the runbooks as a leading indicator of storage performance degradation, with the relief valves that can be tuned
   • Keep tracking down the storage-engine reference leak
   • Move the disk check out of the health path

   Lessons

   Because the storage process uses little memory and the page cache takes what is left, a page cache too small for the data being queried does not show up as memory pressure on the usual memory metrics. InfluxData now treats major page faults as a leading indicator of storage performance degradation and alerts on cache pressure before customers see it.

   View case
2. IL-0305 Replicas of one Intercom Vitess shard hit 100% CPU and fell up to 2.2 hours behind, serving stale inbox data

   Intercom2026

   Symptoms

   From 17:03 UTC on January 27, 2026, a small subset of Intercom customers, fewer than 1%, saw inconsistent latency and stale data when reading from the platform. At 17:25 UTC an alarm on an excessive message backlog in a worker declared an incident automatically.

   Replication lagDatabase overloadHigh latency

   Trigger

   An internal app deletion process, running more concurrent deletion steps per app than usual with no rate limiting, and the ETL data pipeline both put heavy load on the same database shard at the same time.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase replication

   Contributing factor

   No headroom

   Detection

   An alarm on an excessive message backlog in a worker fired at 17:25 UTC and declared an incident automatically.

   Remediation

   • Disabled ETL processing to reduce load on the affected shard
   • Scaled the app deletion workers to zero
   • Re-enabled ETL carefully once monitoring showed stable metrics

   Action items

   • Increase the capacity of shards to leave headroom for unexpected load spikes
   • Add rate limiting and resource controls to the app deletion process so it backs off when database load is high
   • Review how background work such as ETL and maintenance tasks competes with customer-facing operations for database capacity
   • Alert on replication lag earlier and link it to specific workloads
   • Communicate proactively with affected customers in incidents with a limited blast radius

   Lessons

   Because data is spread across many independent shards, the overload stayed on one shard and reached fewer than 1% of customers instead of the whole platform.

   View case
3. IL-0376 A maintenance update set off needless full syncs between Upstash Redis replicas in eu-west-1, slowing the databases

   Upstash2025

   Symptoms

   On 25 February 2025, regional Upstash Redis databases in AWS eu-west-1 slowed down after a maintenance update, with very high CPU load on databases that have a replica in that region. Latency stayed high until about 21:23 UTC, even after the affected databases were unthrottled at 15:06 UTC.

   High latencySustained high node CPU

   Trigger

   A maintenance update to the regional Upstash Redis databases in AWS eu-west-1.

   Root cause

   Software defect

   Component

   In-memory cacheDatabase replication

   Contributing factor

   Automation amplification

   Remediation

   • Added more resources to the regional eu-west-1 cluster
   • Unthrottled the affected databases at 15:06 UTC and enabled more throughput

   Action items

   • Deploy a fix that stops regional databases from starting unnecessary full synchronisations
   • Contact regional database users about migrating to Upstash Global, which does not have this issue

   View case
4. IL-0308 Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slow

   RevenueCat2022

   Symptoms

   At 9:34 UTC on November 23, 2022, RevenueCat started moving traffic from its Aurora PostgreSQL 10.x cluster to a new Aurora PostgreSQL 14.x cluster. By 9:39 UTC all traffic was on the new cluster, latency was bad, and the RevenueCat dashboard and the endpoints used by the RevenueCat SDK stopped working.

   Database overloadHigh latencyService outageConnection pool exhaustion

   Trigger

   At 9:34 UTC production traffic was switched from the Aurora PostgreSQL 10.x cluster to the new Aurora PostgreSQL 14.x cluster, before ANALYZE had run on its largest tables.

   Root cause

   Breaking change in an upgrade

   Component

   DatabasePostgreSQL vacuumDatabase replicationDatabase proxy and connection poolerQuery planner and table statistics

   Contributing factor

   Unrepresentative stagingUnverified assumptionAutomation amplificationMonitoring blind spot

   Detection

   At 9:39 UTC, with all traffic on the new cluster, latency looked bad and the infrastructure team began to investigate.

   Remediation

   • Added a statement timeout to the database user so queries could not hold a connection too long
   • Shed 90% of traffic by returning 503s from the load balancer
   • Dropped indexes that the planner was using wrongly and that were not needed
   • Stopped accepting Apple webhooks for a time
   • Rolled out a release without New Relic to stop its EXPLAIN queries
   • Ran ANALYZE by hand on the biggest tables
   • Raised the values of the affected primary key sequences with a prepared script
   • Stopped the integration event dispatcher by hand
   • Replayed the webhooks that were dispatched but not recorded
   • Restored integration event dispatching, ETLs and other non-web server jobs

   Action items

   • Consider change freezes during business-critical periods
   • Explain better how the SDK behaves during backend outages
   • Tell customers about maintenance and outages in the developer dashboard and by direct email to technical contacts
   • Replay a subset of write traffic in migration dry runs to mimic the cutover
   • Move costly processes to asynchronous work to keep the API fast and stable
   • Re-evaluate table indexes, including unused ones the database may choose for queries
   • Show the load of each endpoint and system in the database dashboards
   • Involve support and customer communications staff during outages

   Lessons

   The new cluster had not been analyzed, and the tests missed it because the test cluster had been up long enough to gather statistics. Dry runs replayed no write traffic, on the belief that writes are simpler than reads, and the team assumed sequence state was replicated with the data. Rolling back at the start would have been better than fixing forward. The migration, timed to finish before Thanksgiving and Black Friday, should have been delayed until the week after, because the US had entered a sensitive period and part of the US support team was already on holiday.

   View case
5. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicas

   Cloudflare2020

   Symptoms

   On 2 November 2020 the Cloudflare API and dashboard were degraded for six hours and 33 minutes. The success rate of API queries dropped at times to 75%, and the dashboard was up to 80 times slower than normal, while Cloudflare's edge network kept working.

   High error rateHigh latencyDatabase overloadWrites rejectedPacket loss

   Trigger

   At 14:43 UTC a network switch in a control plane data center partly failed: it kept LACP and BGP running but stopped forwarding some packets, and it recovered on its own six minutes later.

   Root cause

   DNS or network fault

   Component

   NIC, cable and switch port linkCoordination and key-value storeDatabase failover managerDatabaseDatabase replication

   Contributing factor

   Automation amplificationNo headroomHarmful manual action

   Detection

   At 14:43 UTC alerts fired because the misbehaving switch was unreachable to pings. Engineers then saw that the primary authentication database was overwhelmed.

   Remediation

   • Shed load from the primary authentication database by slowing SSL certificate pushes to the edge, emails and other features
   • Moved API traffic that could use read replicas to the secondary data center by hand
   • Sent authentication calls fully back to the primary data center when the dashboard got worse
   • Rebuilt the first database replica, which put itself back into service at 21:20 UTC and returned performance to normal
   • Ramped the turned-down services back up so asynchronous processing could catch up

   Action items

   • Remove the cluster management limitation that requires rebuilding all replicas after a primary is promoted
   • Redesign the user sessions system so dashboard traffic can be steered between data centers
   • Change the auto-remediation configuration so a database replica is not promoted so quickly
   • Tune each system involved to be more resilient to failures of its dependencies

   Lessons

   Every system in the chain had redundancy and none failed completely; each only degraded, which made the chain of events much harder to model and foresee. Automating replica promotion so that it could trigger on a minute's notice had been a point of pride, but for at least one database a fast promotion can do more harm than the failure it answers. Most general-purpose cluster managers use Raft or Paxos rather than Byzantine fault tolerant protocols, accepting a rare failure mode in exchange for a protocol that is simpler to implement correctly and debug.

   View case

By root cause7