Incident Library

Organisation

Devsisters

Devsisters · devsisters

2 cases · 2021 to 2024

Root cause Cloud provider outage1Software defect1
Component cert-manager1Application database1Helm1
Contributing factor Unverified assumption1Excessive blast radius1
Symptom Data loss1Node NotReady1Service outage1TLS certificate errors1

Cases

  1. IL-0173 cert-manager's preferredChain logic picked the Let's Encrypt chain through the expired DST Root CA X3, so clients built on an old BoringSSL could not connect to a development serverDevsisters · 2024

    Root cause Software defect

  2. IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configuredDevsisters · 2021

    Root cause Cloud provider outage