Incident Library
Applied filters Root causeSoftware defectSymptomService outageOrganisationincident.ioClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0329 A routine migration left PGAudit stuck holding Postgres locks, causing intermittent timeouts and a 2-minute outage
    incident.io2025
    Symptoms

    On 9 April 2025 from 14:16 UTC, incident.io's dashboard, mobile app, Slack app and API became intermittently slow and requests timed out. Monitoring alerted the team soon after a routine migration began on the production Postgres database, and the incident ended with the primary database unavailable for just under 2 minutes, from 14:25 to 14:27 UTC.

    Lock contention or deadlockIntermittent timeoutsHigh latencyService outage
    Trigger

    At 14:16 UTC a routine migration that created an empty table and added an index ran on the production database, where PGAudit had been re-enabled after the weekend upgrade to Postgres 17.

    Root cause
    Software defect
    Component
    Database
    Contributing factor
    –
    Detection

    incident.io's monitoring alerted the team to the problem shortly after the migration started.

    Remediation
    • Restarted the primary database to force the stuck PGAudit process to end, taking just under 2 minutes of downtime
    • Disabled and then fully removed the PGAudit extension after service was restored
    Action items
    • Keep PGAudit removed until its safety can be guaranteed
    • Continue investigating the interaction between the migration and PGAudit
    • Hold an internal debrief of the incident
    • Improve processes and monitoring to detect, prevent and respond to similar situations
    View case
Check your cluster for this

This case involves Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
Database
Contributing factor
–
Organisation
incident.io
  1. IL-0329 A routine migration left PGAudit stuck holding Postgres locks, causing intermittent timeouts and a 2-minute outage
    incident.io2025