Cases1 case

1 to 1 of 1

1. IL-0329 A routine migration left PGAudit stuck holding Postgres locks, causing intermittent timeouts and a 2-minute outage

   incident.io2025

   Symptoms

   On 9 April 2025 from 14:16 UTC, incident.io's dashboard, mobile app, Slack app and API became intermittently slow and requests timed out. Monitoring alerted the team soon after a routine migration began on the production Postgres database, and the incident ended with the primary database unavailable for just under 2 minutes, from 14:25 to 14:27 UTC.

   Lock contention or deadlockIntermittent timeoutsHigh latencyService outage

   Trigger

   At 14:16 UTC a routine migration that created an empty table and added an index ran on the production database, where PGAudit had been re-enabled after the weekend upgrade to Postgres 17.

   Root cause

   Software defect

   Component

   Database

   Contributing factor

   –

   Detection

   incident.io's monitoring alerted the team to the problem shortly after the migration started.

   Remediation

   • Restarted the primary database to force the stuck PGAudit process to end, taking just under 2 minutes of downtime
   • Disabled and then fully removed the PGAudit extension after service was restored

   Action items

   • Keep PGAudit removed until its safety can be guaranteed
   • Continue investigating the interaction between the migration and PGAudit
   • Hold an internal debrief of the incident
   • Improve processes and monitoring to detect, prevent and respond to similar situations

   View case

By root cause1