Incident Library

Organisation

Reddit

Reddit · reddit

1 case · 2023

Root cause Breaking change in an upgrade
Component Admission webhookCalicoetcdKubernetes API server
Contributing factor Untested backupConfiguration not in codeMonitoring blind spotSingle point of failureUnrepresentative staging
Symptom Control plane lagDNS resolution failuresHigh error rateMonitoring outagePods stuck in ContainerCreatingService outage

Cases

  1. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutesReddit · 2023

    Root cause Breaking change in an upgrade