Incident Library
Applied filters Root causeFaulty deploymentComponentSchema migrationSymptomDatabase overloadClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0369 Railway's backend API went down when a column migration waited on locks and replicas could not get database connections
    Railway2025
    Symptoms

    At 15:18 UTC on 9 December 2025, Railway's backend replicas became unhealthy and stopped serving traffic. Dashboard access, CLI operations, logins, the API and the processing of GitHub-based deployments were disrupted, while running deployments stayed online.

    Probe failuresLock contention or deadlockDatabase overloadService outage
    Trigger

    Engineers merged a database schema change with a data migration at 15:03 UTC, and the migration began applying at 15:17 UTC. It added a nullable column to a heavily used table of about 1 billion rows.

    Root cause
    Faulty deployment
    Component
    DatabaseDatabase proxy and connection poolerSchema migrationLiveness and readiness probes
    Contributing factor
    Health check tied to a dependency
    Action items
    • Increase database connections and reconfigure PgBouncer to work within those limits
    • Fix the long-running query so it no longer causes row lock issues
    View case
  2. IL-0554 A database migration used up Sentry US's Postgres connections, and the site and event ingestion were down for 22 minutes
    Sentry2024
    Symptoms

    From 3:42 PM to 4:04 PM PT on 10 October 2024, the Sentry US site and event ingestion were unavailable, and the status page reported an issue with the Postgres database that made the site unavailable. The dashboard, the API and every US ingestion type, including errors, transactions, attachments, profiles, replays, spans, cron monitoring and custom metrics, were affected.

    Service outageDatabase overload
    Trigger

    A database migration ran certain database operations through the RunSQL command without lock and statement timeouts.

    Root cause
    Faulty deployment
    Component
    DatabaseSchema migration
    Contributing factor
    No configuration testsNo runbook
    Remediation
    • Issued a fix, after which the dashboard and ingestion recovered
    Action items
    • Replace RunSQL commands with more robust solutions in the migration framework
    • Update the database framework to the latest version, which handles some operations RunSQL was used for
    • Write runbooks for recovering from lock or timeout issues during migrations
    • Improve alerting and observability so the team reacts sooner
    View case
  3. IL-0524 An index migration locked Open Collective's whole Transactions table and degraded the service for nearly an hour
    Open Collective2023
    Symptoms

    At 09:20 UTC on 4 April 2023, seven minutes after a deployment was triggered, Open Collective's service degraded: a migration in the deployment had locked the entire Transactions table in the production database.

    Lock contention or deadlockDatabase overload
    Trigger

    A deployment ran a migration that dropped indexes on the Transactions table and re-created them with CONCURRENTLY, excluding soft-deleted rows.

    Root cause
    Faulty deployment
    Component
    DatabaseSchema migration
    Contributing factor
    Missed in reviewUnrepresentative staging
    Detection

    The degradation of service was noticed at 09:20 UTC, seven minutes after the deployment was triggered.

    Remediation
    • Killed the migration, though the table remained locked
    • Put the frontend into maintenance mode to limit queries
    • Closed the other connections to the database
    • Cancelled the new index creation and restored the old index
    • Restored the frontend and confirmed the issue was resolved
    Action items
    • Add a script that simulates production load locally, using a tool like vegeta
    • Run this kind of migration during specific maintenance windows
    • Create the new index first and drop the old one afterwards in future migrations
    • Insist on a review from another engineer before merging such a pull request
    Lessons

    Dropping the old index before creating the new one was too optimistic under constant production queries, even with CONCURRENTLY, and an unreviewed pull request let it through. Tests locally and on staging did not show the problem because there were no tools to run migrations in stressed environments. Putting the frontend in maintenance mode limited the impact while showing users a clear message.

    View case
Check your cluster for this

These 3 cases involve Database and Schema migration. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment3 cases

Component
Database3Schema migration3Database proxy and connection pooler1Liveness and readiness probes1
Contributing factor
No configuration tests1No runbook1Health check tied to a dependency1Missed in review1Unrepresentative staging1
Organisation
Open Collective1Railway1Sentry1
  1. IL-0369 Railway's backend API went down when a column migration waited on locks and replicas could not get database connections
    Railway2025
  2. IL-0554 A database migration used up Sentry US's Postgres connections, and the site and event ingestion were down for 22 minutes
    Sentry2024
  3. IL-0524 An index migration locked Open Collective's whole Transactions table and degraded the service for nearly an hour
    Open Collective2023