Incident Library

Organisation

GoCardless

GoCardless · gocardless

3 cases · 2015 to 2020

Root cause Certificate or credential expiry1Faulty deployment1Hardware failure1
Component Database2cert-manager1Database failover manager1Schema migration1Secrets store1
Contributing factor Unverified assumption3Harmful manual action1Monitoring blind spot1Overlapping failures1Single point of failure1
Symptom Service outage3Container crash loop1Credentials unavailable1Failover failure1Lock contention or deadlock1TLS certificate errors1

Cases

  1. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outageGoCardless · 2020

    Root cause Certificate or credential expiry

  2. IL-0219 GoCardless's API and Dashboard failed every request for 1 hour 50 minutes when Pacemaker did not promote a replicaGoCardless · 2017

    Root cause Hardware failure

  3. IL-0391 GoCardless API requests timed out for about 15 seconds while a Postgres migration waited for an AccessExclusive lockGoCardless · 2015

    Root cause Faulty deployment