Incident Library

Organisation

Sentry

Sentry · sentry

2 cases · 2024

Root cause Capacity exhaustion1Faulty deployment1
Component Automated capacity manager1CDN and edge network1Database1Schema migration1
Contributing factor Automation amplification1Excessive blast radius1No configuration tests1No runbook1
Symptom Service outage2Autoscaler misbehaviour1Database overload1High error rate1

Cases

  1. IL-0555 Sentry's ingestion edge regions backlogged one after another and rejected traffic in three incidents in October 2024Sentry · 2024

    Root cause Capacity exhaustion

  2. IL-0554 A database migration used up Sentry US's Postgres connections, and the site and event ingestion were down for 22 minutesSentry · 2024

    Root cause Faulty deployment