Incident Library
Applied filters Root causeSoftware defectOrganisationincident.ioClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0329 A routine migration left PGAudit stuck holding Postgres locks, causing intermittent timeouts and a 2-minute outage
    incident.io2025
    Symptoms

    On 9 April 2025 from 14:16 UTC, incident.io's dashboard, mobile app, Slack app and API became intermittently slow and requests timed out. Monitoring alerted the team soon after a routine migration began on the production Postgres database, and the incident ended with the primary database unavailable for just under 2 minutes, from 14:25 to 14:27 UTC.

    Lock contention or deadlockIntermittent timeoutsHigh latencyService outage
    Trigger

    At 14:16 UTC a routine migration that created an empty table and added an index ran on the production database, where PGAudit had been re-enabled after the weekend upgrade to Postgres 17.

    Root cause
    Software defect
    Component
    Database
    Contributing factor
    –
    Detection

    incident.io's monitoring alerted the team to the problem shortly after the migration started.

    Remediation
    • Restarted the primary database to force the stuck PGAudit process to end, taking just under 2 minutes of downtime
    • Disabled and then fully removed the PGAudit extension after service was restored
    Action items
    • Keep PGAudit removed until its safety can be guaranteed
    • Continue investigating the interaction between the migration and PGAudit
    • Hold an internal debrief of the incident
    • Improve processes and monitoring to detect, prevent and respond to similar situations
    View case
  2. IL-0330 Requests timed out for two weeks waiting on Go's database/sql connection pool, one for almost 20 seconds
    incident.io2023
    Symptoms

    Over two weeks in early 2023, incident.io's Go application intermittently timed out on requests that used its Postgres database. A customer reported the problem, Sentry showed a burst of context canceled errors, and the first episode was over within a minute.

    Connection pool exhaustionIntermittent timeouts
    Root cause
    Software defect
    Component
    Database
    Contributing factor
    Monitoring blind spot
    Detection

    A customer report and a flurry of context canceled errors in Sentry; the on-call engineer opened an incident and checked the Grafana API dashboard. In the second episode an incident was opened automatically and the on-call engineer was pulled into it.

    Remediation
    • Moved the policy violations calculation to a materialised view
    • Added database indices for slow queries
    • Rewrote queries to filter on an indexed column already present in the join
    • Added a one second lock timeout to all transactions
    • Forwarded incoming Slack webhooks to PubSub and processed them asynchronously
    • Added a middleware that logs how long each operation holds a database connection
    • Removed the unnecessary transaction from most Slack modal submissions, keeping it only where needed
    Lessons

    The cause was not one slow operation but many short transactions that, added together, caused the problem. The investigation left incident.io with a way to filter logs and traces by how long each operation holds a database connection, and with real performance improvements. Separate databases for separate parts of the application would probably have made the cause easier to find, but would bring distributed transactions and less convenient development environments, a trade-off incident.io chose not to make for now.

    View case
Check your cluster for this

These 2 cases involve Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect2 cases

Component
Database2
Contributing factor
Monitoring blind spot1
Organisation
incident.io2
  1. IL-0329 A routine migration left PGAudit stuck holding Postgres locks, causing intermittent timeouts and a 2-minute outage
    incident.io2025
  2. IL-0330 Requests timed out for two weeks waiting on Go's database/sql connection pool, one for almost 20 seconds
    incident.io2023