Cases6 cases

1 to 5 of 6

1. IL-0379 InfluxDB Cloud queries in Azure West Europe slowed and timed out as storage pods fell behind and kept restarting

   InfluxData2026

   Symptoms

   From the evening of 27 September 2026 (UTC), InfluxDB Cloud customers in Azure West Europe saw slow queries and query timeouts, and a query queue depth alert fired at 22:58 UTC. Writes were accepted the whole time, but new data on some partitions could not be queried right away.

   High latencyIntermittent timeoutsReplication lagProbe failures

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase replicationLiveness and readiness probesPod scheduling and placement

   Contributing factor

   Monitoring blind spotAutomation amplification

   Detection

   A query queue depth alert fired at 22:58 UTC on 27 September and an incident was opened.

   Remediation

   • Doubled query-layer capacity
   • Restarted stuck storage pods one at a time
   • Raised the readiness probe timeout and rolled it out fleet-wide
   • Doubled recovery concurrency after a restart
   • Raised the cleanup timeout to 24 hours, which stopped the forced restarts
   • Raised each storage pod's resource request so each host held fewer pods, doubling page cache memory and disk throughput per pod

   Action items

   • Add warning alerts for page-fault rate, cache coverage, disk reads on storage volumes and query abandonment
   • List major page faults in the runbooks as a leading indicator of storage performance degradation, with the relief valves that can be tuned
   • Keep tracking down the storage-engine reference leak
   • Move the disk check out of the health path

   Lessons

   Because the storage process uses little memory and the page cache takes what is left, a page cache too small for the data being queried does not show up as memory pressure on the usual memory metrics. InfluxData now treats major page faults as a leading indicator of storage performance degradation and alerts on cache pressure before customers see it.

   View case
2. IL-0557 Database transaction locks piled up on Storj's US1 satellite, and uploads and downloads failed with timeouts for 2 hours

   Storj2025

   Symptoms

   From 21:04 UTC on 7 August 2025, database transaction locks on the Storj US1 satellite started to increase, and upload and download requests began to fail. The on-call team was paged at 21:14 UTC.

   Lock contention or deadlockIntermittent timeoutsConnection pool exhaustionHigh error rate

   Trigger

   An unusually high volume of concurrent uploads to the same objects reached the US1 satellite.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   –

   Detection

   The on-call team was paged at 21:14 UTC, ten minutes after database transaction locks began to rise.

   Remediation

   • Reset the state of connections periodically to keep contention errors from growing in a cascade

   Action items

   • Remove the sources of contention between database transactions with code changes
   • Add monitoring and alerts for increased levels of database locks
   • Look for further process and system improvements, including tools that reduce or remove the need for manual maintenance

   View case
3. IL-0544 Wasabi's database cache stopped responding when its queueing service filled up, and S3 failed in all regions

   Wasabi2024

   Symptoms

   From 10:32 UTC on 6 August 2024, Wasabi's S3 service and user services failed in every region: users had trouble logging into the Web Console and requests to Wasabi regions returned errors.

   Service outageFailover failureIntermittent timeouts

   Trigger

   At 10:32 UTC on 6 August, Wasabi's queueing service reached full capacity, and that made the database cache unresponsive.

   Root cause

   Capacity exhaustion

   Component

   Message brokerIn-memory cacheDatabase

   Contributing factor

   –

   Remediation

   • Restarted the primary database to clear stale connections and cleared the queueing service's queue at the same time, which did not fully restore the database
   • Promoted the secondary database instance to primary, after which S3 was fully operational at 11:20 UTC
   • Restarted the database again between 13:17 and 13:23 UTC to fully incorporate the queueing service library
   • Promoted the original primary database back to production on 7 August, which stopped the user service timeouts
   • Recreated and redeployed the automation cluster that the automation service could not see, which fully restored S3

   View case
4. IL-0550 Bitbucket Cloud's database slowed until connection pools saturated and requests timed out, leaving Pipelines pending

   Atlassian2024

   Symptoms

   From 07:22 UTC on 22 February 2024, the bitbucket.org website and the Bitbucket Cloud APIs became slow, and Bitbucket Pipelines runs were badly delayed. Git requests over HTTPS and SSH kept working.

   High latencyConnection pool exhaustionIntermittent timeoutsThroughput degradation

   Root cause

   Capacity exhaustion

   Component

   DatabasePostgreSQL vacuum

   Contributing factor

   No alert on limit

   Remediation

   • Added resources to Bitbucket Pipelines to clear the backlog of jobs, which was fully cleared by 17:30 UTC

   Action items

   • Reconfigure the vacuuming threshold for database tables with high write activity
   • Adjust alert thresholds to catch this behaviour earlier
   • Tune autoscaling and load shedding for the Pipelines services and increase build runner capacity

   View case
5. IL-0386 After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodes

   incident.io2023

   Symptoms

   Shortly after incident.io moved its infrastructure fully into Google Cloud, the application saw a larger-than-normal number of Postgres connection timeouts, and new database connections per second were in the hundreds, spiking up and down. At the same time, the memcached instance used for transient data and faster API responses had sporadic connection timeouts.

   Intermittent timeoutsPacket loss

   Trigger

   After the move to Google Cloud, the application opened hundreds of new Postgres connections per second, and an accidental database join made it send thousands of duplicate calls to an external third party in parallel.

   Root cause

   Capacity exhaustion

   Component

   CNI pluginDatabaseIn-memory cache

   Contributing factor

   Unreviewed defaultsOverlapping failures

   Detection

   An incident was declared when the number of Postgres connection timeouts rose above normal. The remaining cache timeouts were traced through the logs of the Kubernetes node that served an affected request, the request's trace and the node's metrics.

   Remediation

   • Doubled the maximum Postgres connection lifespan from 15m to 30m
   • Made the Postgres connection pools static so they always keep N connections open
   • Raised the memcache client's MaxIdleConns from 2 to 20
   • Moved temporarily from Google's hosted memcached to a memcached inside the Kubernetes cluster
   • Shipped the fix for the accidental database join that made thousands of duplicate external calls

   Action items

   • Set more sensible concurrency limits for outbound network calls

   Lessons

   Network timeouts are not always local to one request or pod: when they appear, the whole activity of the node, including other pods on it, needs to be checked. Staying within a provider's rate limits does not keep outbound calls from affecting a node, so outbound network calls need more sensible concurrency limits.

   View case

By root cause1