Incident Library

Organisation

incident.io

incident.io · incident-io

3 cases · 2023 to 2025

Root cause Software defect2Capacity exhaustion1
Component Database3In-memory cache1CNI plugin1
Contributing factor Unreviewed defaults1Monitoring blind spot1Overlapping failures1
Symptom Intermittent timeouts3Connection pool exhaustion1High latency1Lock contention or deadlock1Packet loss1Service outage1

Cases

  1. IL-0329 A routine migration left PGAudit stuck holding Postgres locks, causing intermittent timeouts and a 2-minute outageincident.io · 2025

    Root cause Software defect

  2. IL-0386 After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodesincident.io · 2023

    Root cause Capacity exhaustion

  3. IL-0330 Requests timed out for two weeks waiting on Go's database/sql connection pool, one for almost 20 secondsincident.io · 2023

    Root cause Software defect