Incident Library

Organisation

Jetstack

Jetstack · jetstack

1 case · 2019

Root cause Configuration change
Component Admission webhookGKE control planeKubernetes API serverkubelet
Contributing factor Automation amplificationExcessive blast radiusMonitoring blind spotNo health probes
Symptom Control plane unresponsiveNode recreation loopService outageUpgrade stall

Cases

  1. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every nodeJetstack · 2019

    Root cause Configuration change