Incident Library

Organisation

Target

Target · target

2 cases · 2018 to 2019

Root cause Configuration change1Dependency failure1
Component Container runtime1Credential broker for pods1Jobs and CronJobs1Node pool1Service discovery1Telemetry agent1
Contributing factor Automation amplification2Excessive blast radius1Unreviewed defaults1Unpatched known bug1Single point of failure1
Symptom Credentials unavailable1High latency1Node NotReady1Service outage1Load imbalance1

Cases

  1. IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip meshTarget · 2019 · sev1

    Root cause Dependency failure

  2. IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a clusterTarget · 2018

    Root cause Configuration change