Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

Untested backup

미검증 백업 · backup-unverified

2 cases · 2021 to 2023

Root cause Configuration change1 Breaking change in an upgrade1

Component Admission webhook1 Argo CD1 Calico1 Configuration templating1 etcd1

Symptom Service outage2 Control plane lag1 DNS resolution failures1 High error rate1

Organisation Reddit1 Skyscanner1

Cases

IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutesReddit · 2023
Root cause Breaking change in an upgrade
IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every regionSkyscanner · 2021
Root cause Configuration change

View on the graph

Kubernetes API server1

Monitoring outage1

Pods stuck in ContainerCreating1

Resources deleted1