Incident Library

Organisation

Zalando

Zalando · zalando

4 cases · 2018 to 2019

Root cause Capacity exhaustion3Software defect1
Component Kubernetes API server3CI system1Cloud load balancer1CoreDNS1Credential broker for pods1Client-side DNS resolver and cache1Ingress controller1kubelet1Container memory limit1
Contributing factor Excessive blast radius2Unreviewed defaults2Single point of failure2Unpatched known bug1Unknown limit1Monitoring blind spot1No caching1Retry storm1
Symptom High error rate3OOMKilled2Control plane lag1Control plane unresponsive1Credentials unavailable1DNS resolution failures1Monitoring outage1Node NotReady1

Cases

  1. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limitZalando · 2019

    Root cause Capacity exhaustion

  2. IL-0030 Most nodes of a cluster went NotReady together for 15 minutes while the API server sat behind an AWS ELBZalando · 2019

    Root cause Software defect

  3. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019

    Root cause Capacity exhaustion

  4. IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifestZalando · 2018

    Root cause Capacity exhaustion