Cases3 cases

1 to 3 of 3

1. IL-0329 A routine migration left PGAudit stuck holding Postgres locks, causing intermittent timeouts and a 2-minute outage

   incident.io2025

   Symptoms

   On 9 April 2025 from 14:16 UTC, incident.io's dashboard, mobile app, Slack app and API became intermittently slow and requests timed out. Monitoring alerted the team soon after a routine migration began on the production Postgres database, and the incident ended with the primary database unavailable for just under 2 minutes, from 14:25 to 14:27 UTC.

   Lock contention or deadlockIntermittent timeoutsHigh latencyService outage

   Trigger

   At 14:16 UTC a routine migration that created an empty table and added an index ran on the production database, where PGAudit had been re-enabled after the weekend upgrade to Postgres 17.

   Root cause

   Software defect

   Component

   Database

   Contributing factor

   –

   Detection

   incident.io's monitoring alerted the team to the problem shortly after the migration started.

   Remediation

   • Restarted the primary database to force the stuck PGAudit process to end, taking just under 2 minutes of downtime
   • Disabled and then fully removed the PGAudit extension after service was restored

   Action items

   • Keep PGAudit removed until its safety can be guaranteed
   • Continue investigating the interaction between the migration and PGAudit
   • Hold an internal debrief of the incident
   • Improve processes and monitoring to detect, prevent and respond to similar situations

   View case
2. IL-0386 After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodes

   incident.io2023

   Symptoms

   Shortly after incident.io moved its infrastructure fully into Google Cloud, the application saw a larger-than-normal number of Postgres connection timeouts, and new database connections per second were in the hundreds, spiking up and down. At the same time, the memcached instance used for transient data and faster API responses had sporadic connection timeouts.

   Intermittent timeoutsPacket loss

   Trigger

   After the move to Google Cloud, the application opened hundreds of new Postgres connections per second, and an accidental database join made it send thousands of duplicate calls to an external third party in parallel.

   Root cause

   Capacity exhaustion

   Component

   CNI pluginDatabaseIn-memory cache

   Contributing factor

   Unreviewed defaultsOverlapping failures

   Detection

   An incident was declared when the number of Postgres connection timeouts rose above normal. The remaining cache timeouts were traced through the logs of the Kubernetes node that served an affected request, the request's trace and the node's metrics.

   Remediation

   • Doubled the maximum Postgres connection lifespan from 15m to 30m
   • Made the Postgres connection pools static so they always keep N connections open
   • Raised the memcache client's MaxIdleConns from 2 to 20
   • Moved temporarily from Google's hosted memcached to a memcached inside the Kubernetes cluster
   • Shipped the fix for the accidental database join that made thousands of duplicate external calls

   Action items

   • Set more sensible concurrency limits for outbound network calls

   Lessons

   Network timeouts are not always local to one request or pod: when they appear, the whole activity of the node, including other pods on it, needs to be checked. Staying within a provider's rate limits does not keep outbound calls from affecting a node, so outbound network calls need more sensible concurrency limits.

   View case
3. IL-0330 Requests timed out for two weeks waiting on Go's database/sql connection pool, one for almost 20 seconds

   incident.io2023

   Symptoms

   Over two weeks in early 2023, incident.io's Go application intermittently timed out on requests that used its Postgres database. A customer reported the problem, Sentry showed a burst of context canceled errors, and the first episode was over within a minute.

   Connection pool exhaustionIntermittent timeouts

   Root cause

   Software defect

   Component

   Database

   Contributing factor

   Monitoring blind spot

   Detection

   A customer report and a flurry of context canceled errors in Sentry; the on-call engineer opened an incident and checked the Grafana API dashboard. In the second episode an incident was opened automatically and the on-call engineer was pulled into it.

   Remediation

   • Moved the policy violations calculation to a materialised view
   • Added database indices for slow queries
   • Rewrote queries to filter on an indexed column already present in the join
   • Added a one second lock timeout to all transactions
   • Forwarded incoming Slack webhooks to PubSub and processed them asynchronously
   • Added a middleware that logs how long each operation holds a database connection
   • Removed the unnecessary transaction from most Slack modal submissions, keeping it only where needed

   Lessons

   The cause was not one slow operation but many short transactions that, added together, caused the problem. The investigation left incident.io with a way to filter logs and traces by how long each operation holds a database connection, and with real performance improvements. Separate databases for separate parts of the application would probably have made the cause easier to find, but would bring distributed transactions and less convenient development environments, a trade-off incident.io chose not to make for now.

   View case

By root cause2