Incident Library

Organisation

Stripe

Stripe · stripe

1 case · 2019

Root cause Software defect
Component DatabaseDatabase failover manager
Contributing factor Unverified assumptionExcessive blast radiusChange with a delayed effectHarmful manual actionMonitoring blind spotOverlapping failures
Symptom Failover failureHigh error rateIntermittent timeoutsSustained high node CPUWrites rejected

Cases

  1. IL-0241 A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one dayStripe · 2019

    Root cause Software defect