Incident Library

Organisation

Kubernetes project

Kubernetes project · kubernetes-project

3 cases · 2019

Root cause Faulty deployment2Capacity exhaustion1
Component CI system3Cloud block storage1kubelet1Node root disk1Node pool1
Contributing factor Unverified assumption1Change started before off-hours1Change with a delayed effect1Ignored early signal1Limited debug access1Monitoring blind spot1No alert on limit1No resource requests or limits1Single point of failure1Unrepresentative staging1
Symptom Container crash loop1High latency1Sustained high node CPU1Node kernel panic1Node NotReady1Pods Pending1Pods stuck in Terminating1Service outage1Throughput degradation1

Cases

  1. IL-0401 Prow jobs on the k8s-prow-build GKE cluster failed and merges stalled as nodes hit IOPS throttling and kernel panicsKubernetes project · 2019

    Root cause Capacity exhaustion

  2. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hoursKubernetes project · 2019

    Root cause Faulty deployment

  3. IL-0466 Prow scheduled test job pods onto its service cluster, leaving hundreds Pending and starving Prow's own servicesKubernetes project · 2019

    Root cause Faulty deployment