Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Pod eviction

파드 축출 · pods-evicted

7 cases · 2018 to 2025

Root cause Configuration change3 Capacity exhaustion2 Faulty deployment1 Software defect1

Component Node pool5 AWS VPC CNI2 Cluster autoscaler1 CoreDNS1

Contributing factor No headroom3 Excessive blast radius2 No alert on limit2 Missed in review2

Organisation 2i2c2 UK Ministry of Justice2 CoW Protocol1 Grafana Labs1 Oath

Cases

IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes2i2c · 2025
Root cause Capacity exhaustion
IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went downUK Ministry of Justice · 2025
Root cause Faulty deployment
IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours

View on the graph

Kubernetes controller manager1

Node launch template1

Container memory limit1

Node root disk1

Pod priority and preemption1

Automation amplification1

Unreviewed defaults1

Monitoring blind spot1

No configuration tests1

No health probes1

No resource requests or limits1

Overlapping failures1

Partial rollout1

Unpinned version1

1

CoW Protocol · 2024

Root cause Configuration change

IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on deviceUK Ministry of Justice · 2023

Root cause Configuration change

IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023

Root cause Capacity exhaustion

IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus downGrafana Labs · 2019

Root cause Configuration change

IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod evictionOath · 2018

Root cause Software defect