Incident Library

Applied filters SymptomLock contention or deadlockClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

Cases21 cases

6 to 10 of 21

IL-0369 Railway's backend API went down when a column migration waited on locks and replicas could not get database connections
Railway2025
Symptoms
At 15:18 UTC on 9 December 2025, Railway's backend replicas became unhealthy and stopped serving traffic. Dashboard access, CLI operations, logins, the API and the processing of GitHub-based deployments were disrupted, while running deployments stayed online.
Probe failures Lock contention or deadlock Database overload Service outage
Trigger
Engineers merged a database schema change with a data migration at 15:03 UTC, and the migration began applying at 15:17 UTC. It added a nullable column to a heavily used table of about 1 billion rows.
Root cause
Faulty deployment
Component
Database Database proxy and connection pooler Schema migration Liveness and readiness probes
Contributing factor
Health check tied to a dependency
Action items
Increase database connections and reconfigure PgBouncer to work within those limits
Fix the long-running query so it no longer causes row lock issues
View case
IL-0315 Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connection
Railway2025
Symptoms
At 18:36 UTC on 28 October 2025, two minutes after a database change went live in production, Railway's internal monitoring alerted and paged the engineering team. The dashboard could not be opened, CLI operations that use the API, such as railway up, failed, GitHub-based deployments were delayed because webhook processing failed, and all Railway Public API operations were disrupted.
Service outage Lock contention or deadlock Database overload
Trigger
A routine schema change to the backend's Postgres database, introduced at 18:23 UTC and live in production at 18:34 UTC, that added a new column with an index without the CONCURRENTLY option.
Root cause
Faulty deployment
Component
Database Database proxy and connection pooler
Contributing factor
No configuration tests Limited debug access
Detection
Internal monitoring alerts paged the engineering team at 18:36 UTC, and the incident was declared at 18:41 UTC.
Remediation
The index migration finished after about 30 minutes and released the table lock, and the queued operations and connection pools recovered
Action items
Enforce CONCURRENTLY for all index creation in CI and block pull requests that do not use it
Adjust PgBouncer pool limits so they cannot overwhelm the Postgres instance's capacity
Configure database user connection limits so administrative access stays possible during incidents
View case
IL-0436 A supremum lock left by ROLLBACK TO SAVEPOINT in a long transaction blocked inserts and filled Aurora MySQL connections
freee2025
Symptoms
Connections to Aurora MySQL failed again and again in one freee service until its web pages could not be opened at all. The database did not respond, the web application timed out and could not open new database connections, and the whole service was down.
Lock contention or deadlock Service outage
Trigger
A long transaction inserted a row into the table after a SAVEPOINT and then ran ROLLBACK TO SAVEPOINT, which left it holding an exclusive lock on the supremum of the table's primary index.
Root cause
Software defect
Component
Database
Contributing factor
Unverified assumption Unreviewed defaults
Action items
As a rule stop issuing SAVEPOINT, split a transaction when a partial rollback is needed, and set an application-side timeout where a nested transaction is unavoidable
Teach developers to consider READ COMMITTED per session for use cases that can accept phantom rows
Monitor long transactions and keep working to remove them across the company
Lessons
ROLLBACK TO SAVEPOINT rolls back rows but does not release their locks, so rolling back one inserted row can leave a supremum lock that blocks every INSERT into the table. ORM frameworks, including Active Record for nested transactions, use SAVEPOINT internally without developers noticing. Locks held by long transactions often go unnoticed when few workloads are blocked, and this incident showed their hidden risk.
View case
IL-0557 Database transaction locks piled up on Storj's US1 satellite, and uploads and downloads failed with timeouts for 2 hours
Storj2025
Symptoms
From 21:04 UTC on 7 August 2025, database transaction locks on the Storj US1 satellite started to increase, and upload and download requests began to fail. The on-call team was paged at 21:14 UTC.
Lock contention or deadlock Intermittent timeouts Connection pool exhaustion High error rate
Trigger
An unusually high volume of concurrent uploads to the same objects reached the US1 satellite.
Root cause
Capacity exhaustion
Component
Database
Contributing factor
–
Detection
The on-call team was paged at 21:14 UTC, ten minutes after database transaction locks began to rise.
Remediation
Reset the state of connections periodically to keep contention errors from growing in a cascade
Action items
Remove the sources of contention between database transactions with code changes
Add monitoring and alerts for increased levels of database locks
Look for further process and system improvements, including tools that reduce or remove the need for manual maintenance
View case
IL-0329 A routine migration left PGAudit stuck holding Postgres locks, causing intermittent timeouts and a 2-minute outage
incident.io2025
Symptoms
On 9 April 2025 from 14:16 UTC, incident.io's dashboard, mobile app, Slack app and API became intermittently slow and requests timed out. Monitoring alerted the team soon after a routine migration began on the production Postgres database, and the incident ended with the primary database unavailable for just under 2 minutes, from 14:25 to 14:27 UTC.
Lock contention or deadlock Intermittent timeouts High latency Service outage
Trigger
At 14:16 UTC a routine migration that created an empty table and added an index ran on the production database, where PGAudit had been re-enabled after the weekend upgrade to Postgres 17.
Root cause
Software defect
Component
Database
Contributing factor
–
Detection
incident.io's monitoring alerted the team to the problem shortly after the migration started.
Remediation
Restarted the primary database to force the stuck PGAudit process to end, taking just under 2 minutes of downtime
Disabled and then fully removed the PGAudit extension after service was restored
Action items
Keep PGAudit removed until its safety can be guaranteed
Continue investigating the interaction between the migration and PGAudit
Hold an internal debrief of the incident
Improve processes and monitoring to detect, prevent and respond to similar situations
View case

Check your cluster for this

These 21 cases involve Database and Schema migration. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap