Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0554 A database migration used up Sentry US's Postgres connections, and the site and event ingestion were down for 22 minutes
    Sentry2024
    Symptoms

    From 3:42 PM to 4:04 PM PT on 10 October 2024, the Sentry US site and event ingestion were unavailable, and the status page reported an issue with the Postgres database that made the site unavailable. The dashboard, the API and every US ingestion type, including errors, transactions, attachments, profiles, replays, spans, cron monitoring and custom metrics, were affected.

    Service outageDatabase overload
    Trigger

    A database migration ran certain database operations through the RunSQL command without lock and statement timeouts.

    Root cause
    Faulty deployment
    Component
    DatabaseSchema migration
    Contributing factor
    No configuration testsNo runbook
    Remediation
    • Issued a fix, after which the dashboard and ingestion recovered
    Action items
    • Replace RunSQL commands with more robust solutions in the migration framework
    • Update the database framework to the latest version, which handles some operations RunSQL was used for
    • Write runbooks for recovering from lock or timeout issues during migrations
    • Improve alerting and observability so the team reacts sooner
    View case
Check your cluster for this

This case involves Database and Schema migration. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment1 case

Component
DatabaseSchema migration
Contributing factor
No configuration testsNo runbook
Organisation
Sentry
  1. IL-0554 A database migration used up Sentry US's Postgres connections, and the site and event ingestion were down for 22 minutes
    Sentry2024