Cases21 cases

9 to 13 of 21

1. IL-0557 Database transaction locks piled up on Storj's US1 satellite, and uploads and downloads failed with timeouts for 2 hours

   Storj2025

   Symptoms

   From 21:04 UTC on 7 August 2025, database transaction locks on the Storj US1 satellite started to increase, and upload and download requests began to fail. The on-call team was paged at 21:14 UTC.

   Lock contention or deadlockIntermittent timeoutsConnection pool exhaustionHigh error rate

   Trigger

   An unusually high volume of concurrent uploads to the same objects reached the US1 satellite.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   –

   Detection

   The on-call team was paged at 21:14 UTC, ten minutes after database transaction locks began to rise.

   Remediation

   • Reset the state of connections periodically to keep contention errors from growing in a cascade

   Action items

   • Remove the sources of contention between database transactions with code changes
   • Add monitoring and alerts for increased levels of database locks
   • Look for further process and system improvements, including tools that reduce or remove the need for manual maintenance

   View case
2. IL-0329 A routine migration left PGAudit stuck holding Postgres locks, causing intermittent timeouts and a 2-minute outage

   incident.io2025

   Symptoms

   On 9 April 2025 from 14:16 UTC, incident.io's dashboard, mobile app, Slack app and API became intermittently slow and requests timed out. Monitoring alerted the team soon after a routine migration began on the production Postgres database, and the incident ended with the primary database unavailable for just under 2 minutes, from 14:25 to 14:27 UTC.

   Lock contention or deadlockIntermittent timeoutsHigh latencyService outage

   Trigger

   At 14:16 UTC a routine migration that created an empty table and added an index ran on the production database, where PGAudit had been re-enabled after the weekend upgrade to Postgres 17.

   Root cause

   Software defect

   Component

   Database

   Contributing factor

   –

   Detection

   incident.io's monitoring alerted the team to the problem shortly after the migration started.

   Remediation

   • Restarted the primary database to force the stuck PGAudit process to end, taking just under 2 minutes of downtime
   • Disabled and then fully removed the PGAudit extension after service was restored

   Action items

   • Keep PGAudit removed until its safety can be guaranteed
   • Continue investigating the interaction between the migration and PGAudit
   • Hold an internal debrief of the incident
   • Improve processes and monitoring to detect, prevent and respond to similar situations

   View case
3. IL-0556 A transaction left open during database maintenance stopped uploads and then downloads on Storj's US1 satellite

   Storj2024

   Symptoms

   At 14:56 UTC on 20 May 2024, write operations on the Storj US1 satellite began to fail, so uploads stopped working, and downloads failed after them. The on-call team was paged at 14:59 UTC, while the AP1 and EU1 satellites kept working normally.

   Lock contention or deadlockService outage

   Trigger

   During routine, manual database maintenance on the US1 satellite that began at 14:55 UTC, a transaction was left open by mistake while other work was being done.

   Root cause

   Operator error

   Component

   Database

   Contributing factor

   –

   Detection

   The on-call team received a page at 14:59 UTC, three minutes after writes started failing.

   Remediation

   • Found and closed the open transaction at 15:22 UTC, after which the satellite recovered

   Action items

   • Review and update database maintenance procedures and training so that every transaction is closed before moving on to other tasks
   • Add monitoring and alerts for open transactions that last longer than a set duration
   • Look for further process and system improvements, including tools that reduce or remove the need for manual maintenance

   View case
4. IL-0217 Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outage

   Honeycomb2023

   Symptoms

   Hours after a routine switch between two clusters of Retriever, Honeycomb's storage and query engine, late on 24 July 2023, the internal SLO for Shepherd, the ingest service, began to burn slowly. The performance was only marginally slow and inconsistent, with no good explanation, so engineers treated it as minor and left it for the morning.

   High latencyDatabase overloadLock contention or deadlockService outage

   Trigger

   Late on 24 July Honeycomb switched between two Retriever clusters to avoid a subtle bug, and the new cluster stopped writing the last-written timestamps the schema cache depends on. On the morning of 25 July a feature flag flip meant to move those writes back stopped the remaining writes entirely.

   Root cause

   Software defect

   Component

   DatabaseIn-memory cache

   Contributing factor

   Dependence on a warm cacheIgnored early signalHarmful manual actionUnverified assumption

   Detection

   Hours after the cluster switch, the internal SLO for the ingest service Shepherd started burning slowly. Engineers left it for the morning, when a chart showed that Retriever's database calls updating the last-written records had stopped.

   Remediation

   • Flipped only the writes back to the other Retriever cluster with a feature flag, which did not fix the problem
   • Set up circuit breakers that denied all Shepherd traffic with a 5xx error to protect the database
   • Failed the hard-locked database over to a replica
   • Restarted all Retriever hosts so they could provide data for the cache
   • Set the last-written timestamp of every schema with traffic in the last day to now, so the schema store reloaded the cache
   • Removed the circuit breaker and restarted the query hosts that failed internal checks

   Action items

   • Finish the Retriever cluster migration and remove all code that could disable writes behind the cache
   • Explore ways to strengthen the cache further
   • Lower the contention put on the database during update storms from new schemas
   • Stabilise the performance costs of some operations in the system
   • Study instrumentation and experimentation that detect and handle such edge cases better

   Lessons

   Locally good decisions combined into the outage: the query engine wrote the timestamps because it already saw all the data, the shared cache helped scale ingest and untangle the main database, the cluster switch was done to avoid a bug, frequent deploys hid the feature flag bug, pausing deploys during the investigation kept hosts from restarting, and past near-misses made the team expect a slow failover. Feature flags, frequent deploys, pausing deploys during incidents and learning from near-misses all contributed, although they are among the practices Honeycomb trusts most. For a rare lockup of this kind a database failover is the fastest recovery. Several recent Honeycomb incidents started from trying to avoid a subtle bug.

   View case
5. IL-0483 Replicate's website and API returned slow responses and HTTP 500s for two hours as PostgreSQL connections ran out

   Replicate2023

   Symptoms

   From 05:45 UTC on 11 May 2023, for about two hours, many Replicate customers got slow responses or HTTP 500 errors from the API and the replicate.com website and had trouble running predictions. Monitoring paged an engineer at 05:46 UTC, a minute after the database connection pool became fully saturated.

   High error rateHigh latencyLock contention or deadlock

   Trigger

   A little after 05:00 UTC on 11 May, normal variations in traffic pushed the stacking INSERT queries past their tipping point.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   Change with a delayed effect

   Detection

   Monitoring detected the saturated database connection pool almost immediately and paged an engineer at 05:46 UTC.

   Remediation

   • Switched off all asynchronous prediction update features with a feature flag at 07:31 UTC, after which the database recovered

   Action items

   • Keep the asynchronous prediction update feature disabled and redesign it so it no longer sends stacked INSERTs
   • Review lock contention and other database hotspots that could cause similar problems
   • Publish detailed reports after major incidents

   Lessons

   The system ran in a degraded state that looked successful until it tipped into a full failure. The same asynchronous update work that triggered the outage also partly protected customers who used webhooks, and the stacked INSERTs came directly from a latency improvement: removing 100-150ms brought two updates of one prediction to the database almost together.

   View case

By root cause8