Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Pods stuck in ContainerCreating

파드 ContainerCreating 정체 · pod-stuck-container-creating

3 cases · 2023

Root cause Configuration change2 Breaking change in an upgrade1

Component Node pool2 AWS VPC CNI2 Admission webhook1 Calico1 CoreDNS1

Contributing factor No configuration tests2 Untested backup1 Configuration not in code1 Monitoring blind spot1

Organisation UK Ministry of Justice2 Reddit1

Cases

IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on deviceUK Ministry of Justice · 2023
Root cause Configuration change
IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreatingUK Ministry of Justice · 2023
Root cause Configuration change
IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes

View on the graph

Kubernetes API server1

Node launch template1

Node root disk1

No alert on limit1

Missed in review1

Single point of failure1

Unrepresentative staging1

Reddit · 2023

Root cause Breaking change in an upgrade