Incident Library
Applied filters Root causeFaulty deploymentSymptomLock contention or deadlockClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases7 cases

6 to 7 of 7

  1. IL-0524 An index migration locked Open Collective's whole Transactions table and degraded the service for nearly an hour
    Open Collective2023
    Symptoms

    At 09:20 UTC on 4 April 2023, seven minutes after a deployment was triggered, Open Collective's service degraded: a migration in the deployment had locked the entire Transactions table in the production database.

    Lock contention or deadlockDatabase overload
    Trigger

    A deployment ran a migration that dropped indexes on the Transactions table and re-created them with CONCURRENTLY, excluding soft-deleted rows.

    Root cause
    Faulty deployment
    Component
    DatabaseSchema migration
    Contributing factor
    Missed in reviewUnrepresentative staging
    Detection

    The degradation of service was noticed at 09:20 UTC, seven minutes after the deployment was triggered.

    Remediation
    • Killed the migration, though the table remained locked
    • Put the frontend into maintenance mode to limit queries
    • Closed the other connections to the database
    • Cancelled the new index creation and restored the old index
    • Restored the frontend and confirmed the issue was resolved
    Action items
    • Add a script that simulates production load locally, using a tool like vegeta
    • Run this kind of migration during specific maintenance windows
    • Create the new index first and drop the old one afterwards in future migrations
    • Insist on a review from another engineer before merging such a pull request
    Lessons

    Dropping the old index before creating the new one was too optimistic under constant production queries, even with CONCURRENTLY, and an unreviewed pull request let it through. Tests locally and on staging did not show the problem because there were no tools to run migrations in stressed environments. Putting the frontend in maintenance mode limited the impact while showing users a clear message.

    View case
  2. IL-0391 GoCardless API requests timed out for about 15 seconds while a Postgres migration waited for an AccessExclusive lock
    GoCardless2015
    Symptoms

    Just after a planned Postgres schema migration started, GoCardless received alerts that API requests were timing out. The API was down for about 15 seconds, until the migration went through and the API came back.

    Lock contention or deadlockService outage
    Trigger

    A deploy with a schema migration that renamed the empty invoicing tables and dropped and recreated their foreign key constraints, which referenced some of the most heavily used tables.

    Root cause
    Faulty deployment
    Component
    DatabaseSchema migration
    Contributing factor
    Unverified assumption
    Detection

    Alerts about API requests timing out began just after the schema migration started.

    Lessons

    Avoiding renames of tables in use, table rewrites under an exclusive lock and expensive work while holding one was not enough for GoCardless: a short DDL statement can still cause downtime by waiting in the lock queue behind a long-running query. GoCardless recommends removing long-running queries and transactions from the application and running analytics on an asynchronously updated replica, setting log_min_duration_statement and log_lock_waits to find such queries, setting lock_timeout in migration scripts so a deploy aborts rather than taking the application down, splitting schema changes so locks are held for shorter times, and keeping Postgres up to date. Developers with a console on the production database can cause the same downtime by leaving a transaction open.

    View case
Check your cluster for this

These 7 cases involve Database and Schema migration. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment7 cases

Component
Database7Schema migration4Database proxy and connection pooler2Liveness and readiness probes1
Contributing factor
Unrepresentative staging2Unverified assumption1Change with a delayed effect1Inefficient query1Limited debug access1Harmful manual action1No configuration tests1Health check tied to a dependency1Missed in review1
Organisation
Railway2CrowdWorks1GoCardless1MIT Open Learning1Open Collective1Replicate1
  1. IL-0570 Dropping an unused MySQL table stalled queries on its busy parent table and crowdworks.jp was hard to reach for a minute
    CrowdWorks2026
  2. IL-0510 MIT Learn's RDS PostgreSQL ran out of storage after a release and learn.mit.edu returned 503s and 504s for hours
    MIT Open Learning2026
  3. IL-0369 Railway's backend API went down when a column migration waited on locks and replicas could not get database connections
    Railway2025
  4. IL-0315 Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connection
    Railway2025
  5. IL-0483 Replicate's website and API returned slow responses and HTTP 500s for two hours as PostgreSQL connections ran out
    Replicate2023
  6. IL-0524 An index migration locked Open Collective's whole Transactions table and degraded the service for nearly an hour
    Open Collective2023
  7. IL-0391 GoCardless API requests timed out for about 15 seconds while a Postgres migration waited for an AccessExclusive lock
    GoCardless2015