Incident Library

Component

Secrets store

시크릿 저장소 · secrets-store

3 cases · 2020 to 2023

Root cause Certificate or credential expiry1Cloud provider outage1Configuration change1
Contributing factor Unverified assumption1Automation amplification1Configuration not in code1Unreviewed defaults1Change with a delayed effect1Harmful manual action1Monitoring blind spot1Single point of failure1
Symptom Service outage3Credentials unavailable2Container crash loop1Pods stuck in Terminating1TLS certificate errors1
Organisation GoCardless1MIT Open Learning1Python Software Foundation1

Cases, by severity

  1. IL-0511 MIT xPro stopped responding after a configuration deploy as pgbouncer logins to RDS failed with 'wrong password type'MIT Open Learning · 2023

    Root cause Configuration change

  2. IL-0514 PyPI's backends stopped responding for 7 minutes as rescheduled pods could not start while Consul and Vault were downPython Software Foundation · 2022

    Root cause Cloud provider outage

  3. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outageGoCardless · 2020

    Root cause Certificate or credential expiry

View on the graph