Cases5 cases

1 to 5 of 5

1. IL-0436 A supremum lock left by ROLLBACK TO SAVEPOINT in a long transaction blocked inserts and filled Aurora MySQL connections

   freee2025

   Symptoms

   Connections to Aurora MySQL failed again and again in one freee service until its web pages could not be opened at all. The database did not respond, the web application timed out and could not open new database connections, and the whole service was down.

   Lock contention or deadlockService outage

   Trigger

   A long transaction inserted a row into the table after a SAVEPOINT and then ran ROLLBACK TO SAVEPOINT, which left it holding an exclusive lock on the supremum of the table's primary index.

   Root cause

   Software defect

   Component

   Database

   Contributing factor

   Unverified assumptionUnreviewed defaults

   Action items

   • As a rule stop issuing SAVEPOINT, split a transaction when a partial rollback is needed, and set an application-side timeout where a nested transaction is unavoidable
   • Teach developers to consider READ COMMITTED per session for use cases that can accept phantom rows
   • Monitor long transactions and keep working to remove them across the company

   Lessons

   ROLLBACK TO SAVEPOINT rolls back rows but does not release their locks, so rolling back one inserted row can leave a supremum lock that blocks every INSERT into the table. ORM frameworks, including Active Record for nested transactions, use SAVEPOINT internally without developers noticing. Locks held by long transactions often go unnoticed when few workloads are blocked, and this incident showed their hidden risk.

   View case
2. IL-0329 A routine migration left PGAudit stuck holding Postgres locks, causing intermittent timeouts and a 2-minute outage

   incident.io2025

   Symptoms

   On 9 April 2025 from 14:16 UTC, incident.io's dashboard, mobile app, Slack app and API became intermittently slow and requests timed out. Monitoring alerted the team soon after a routine migration began on the production Postgres database, and the incident ended with the primary database unavailable for just under 2 minutes, from 14:25 to 14:27 UTC.

   Lock contention or deadlockIntermittent timeoutsHigh latencyService outage

   Trigger

   At 14:16 UTC a routine migration that created an empty table and added an index ran on the production database, where PGAudit had been re-enabled after the weekend upgrade to Postgres 17.

   Root cause

   Software defect

   Component

   Database

   Contributing factor

   –

   Detection

   incident.io's monitoring alerted the team to the problem shortly after the migration started.

   Remediation

   • Restarted the primary database to force the stuck PGAudit process to end, taking just under 2 minutes of downtime
   • Disabled and then fully removed the PGAudit extension after service was restored

   Action items

   • Keep PGAudit removed until its safety can be guaranteed
   • Continue investigating the interaction between the migration and PGAudit
   • Hold an internal debrief of the incident
   • Improve processes and monitoring to detect, prevent and respond to similar situations

   View case
3. IL-0217 Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outage

   Honeycomb2023

   Symptoms

   Hours after a routine switch between two clusters of Retriever, Honeycomb's storage and query engine, late on 24 July 2023, the internal SLO for Shepherd, the ingest service, began to burn slowly. The performance was only marginally slow and inconsistent, with no good explanation, so engineers treated it as minor and left it for the morning.

   High latencyDatabase overloadLock contention or deadlockService outage

   Trigger

   Late on 24 July Honeycomb switched between two Retriever clusters to avoid a subtle bug, and the new cluster stopped writing the last-written timestamps the schema cache depends on. On the morning of 25 July a feature flag flip meant to move those writes back stopped the remaining writes entirely.

   Root cause

   Software defect

   Component

   DatabaseIn-memory cache

   Contributing factor

   Dependence on a warm cacheIgnored early signalHarmful manual actionUnverified assumption

   Detection

   Hours after the cluster switch, the internal SLO for the ingest service Shepherd started burning slowly. Engineers left it for the morning, when a chart showed that Retriever's database calls updating the last-written records had stopped.

   Remediation

   • Flipped only the writes back to the other Retriever cluster with a feature flag, which did not fix the problem
   • Set up circuit breakers that denied all Shepherd traffic with a 5xx error to protect the database
   • Failed the hard-locked database over to a replica
   • Restarted all Retriever hosts so they could provide data for the cache
   • Set the last-written timestamp of every schema with traffic in the last day to now, so the schema store reloaded the cache
   • Removed the circuit breaker and restarted the query hosts that failed internal checks

   Action items

   • Finish the Retriever cluster migration and remove all code that could disable writes behind the cache
   • Explore ways to strengthen the cache further
   • Lower the contention put on the database during update storms from new schemas
   • Stabilise the performance costs of some operations in the system
   • Study instrumentation and experimentation that detect and handle such edge cases better

   Lessons

   Locally good decisions combined into the outage: the query engine wrote the timestamps because it already saw all the data, the shared cache helped scale ingest and untangle the main database, the cluster switch was done to avoid a bug, frequent deploys hid the feature flag bug, pausing deploys during the investigation kept hosts from restarting, and past near-misses made the team expect a slow failover. Feature flags, frequent deploys, pausing deploys during incidents and learning from near-misses all contributed, although they are among the practices Honeycomb trusts most. For a rare lockup of this kind a database failover is the fastest recovery. Several recent Honeycomb incidents started from trying to avoid a subtle bug.

   View case
4. IL-0385 GitHub MySQL read replicas hit a semaphore deadlock during a schema migration's table rename and kept crash-recovering

   GitHub2021

   Symptoms

   On 27 November 2021 at 20:40 UTC, a significant part of GitHub's MySQL read replicas entered a semaphore deadlock while a schema migration on a large MySQL table ran its final step, the rename that moves the updated table into place. Core GitHub services, including GitHub Actions, API Requests, Codespaces, Git Operations, Issues, GitHub Packages, GitHub Pages, Pull Requests and Webhooks, became degraded.

   Lock contention or deadlockDatabase overload

   Trigger

   The final rename step of a schema migration on a large MySQL table. Schema migrations are a common task at GitHub and often take weeks to complete.

   Root cause

   Software defect

   Component

   DatabaseSchema migration

   Contributing factor

   No headroomNo staged rollout

   Remediation

   • Promoted every healthy internal replica into the production path to add read capacity
   • Removed production traffic from the broken replicas until each had processed the table rename
   • Moved the recovered replicas back into production to restore capacity

   Action items

   • Continue functional partitioning so migrations can run in canary mode on a single shard
   • Update internal procedures to increase how much each cluster is over-provisioned
   • Pause schema migrations until it is known how to safeguard against this failure
   • Keep investigating the specific failure scenario
   • Test the migration tooling and classify ways to improve it for this kind of scenario

   View case
5. IL-0486 Figma's PostgreSQL 9 database on RDS slowed under high write IOPS and lock contention until the site was unavailable

   Figma2020

   Symptoms

   On January 22, 2020 at 9:45 AM PST, Figma's database metrics showed rising write IOPS and lock contention, while database CPU was lower than normal. An automated alert followed at 10:00 AM, and by 11:42 AM enough API requests had queued up behind the database that Figma became unavailable for some users.

   Lock contention or deadlockDatabase overloadService outage

   Trigger

   An automatic ANALYZE on one of the tables in the query finished at exactly the moment the incident began. The new statistics pushed the planner's row estimate for an inner subquery past 200 rows, and the plan for the whole query switched to an estimate of over 20 million rows.

   Root cause

   Software defect

   Component

   DatabasePostgreSQL vacuum

   Contributing factor

   Overlapping failuresHarmful manual action

   Detection

   Database metrics showed higher write IOPS and lock contention at 9:45 AM PST, and an automated alert at 10:00 AM started the investigation.

   Remediation

   • Cancelled several inessential queries and increased the allocated IOPS
   • Cancelled autovacuum statements running on the largest tables
   • Changed autovacuum_freeze_max_age and restarted the database to stop aggressive autovacuuming
   • Took the service down for emergency maintenance and upgraded the database to PostgreSQL 11

   Action items

   • Revisit transaction timeouts and limits so long-running and expensive queries are terminated
   • Improve metrics, monitoring and database settings for autovacuuming, locking and query performance
   • Audit queries for unnecessary or excessive joins and subqueries that could trip up the query planner
   • Bring the performance tools of the upgraded PostgreSQL into the incident response plans
   • Audit the structure and access patterns of data stored in PostgreSQL and consider moving some of it to datastores suited to those patterns
   • Improve in-app messaging about unexpected maintenance windows
   • Move towards online database upgrades that do not need downtime
   • Let users keep working on already open files safely while offline, even after closing the browser

   Lessons

   Having upgraded staging to PostgreSQL 11, tested it for months and written a production upgrade plan let the team upgrade under poor conditions with confidence. A routine statistics refresh can flip a query to a catastrophic plan, and the same query's plan can differ widely over time, so comparing plans from snapshots taken at different dates exposed the cause.

   View case

By root cause1