Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Pods Pending

파드 Pending · pod-pending

15 cases · 2017 to 2025

Root cause Capacity exhaustion5 Cloud provider outage4 Software defect3 Configuration change1 Dependency failure1 Faulty deployment1

Component Node pool8 Cluster autoscaler6 kubelet3 VPC subnet or pod address range3

Contributing factor Unreviewed defaults5 Monitoring blind spot4 Overlapping failures4 Unverified assumption2

Organisation 2i2c2 loveholidays2 Adevinta1 CoW Protocol1 FREE NOW

Cases

IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletionNeon · 2025
Root cause Capacity exhaustion
IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went downUK Ministry of Justice · 2025
Root cause Faulty deployment
IL-0007 The AWS VPC CNI could not assign IP addresses to pods as an EKS cluster ran out of IPv4 space

View on the graph

Node root disk2

Container runtime1

Application database1

GKE control plane1

Horizontal pod autoscaler1

Ingress controller1

Kubernetes controller manager1

Node operating system image1

Persistent volumes and stateful sets1

Pod scheduling and placement1

Pod termination1

Excessive blast radius2

Ignored early signal2

No capacity planning2

Single point of failure2

Unpatched known bug1

Limited debug access1

Harmful manual action1

No alert on limit1

No health probes1

Pages not delivered1

Ineffective rollback1

Unrepresentative staging1

Unpinned version1

1

UK Ministry of Justice1

Adevinta · 2024

Root cause Capacity exhaustion

IL-0148 A kube-scheduler regression in Kubernetes v1.27.14, v1.28.10 and v1.29.5 made the scheduler panic on a pod whose node affinity named a nonexistent node, so no pods were scheduled on affected AKS clustersMicrosoft · 2024

Root cause Software defect

IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hoursCoW Protocol · 2024

Root cause Configuration change

IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023

Root cause Capacity exhaustion

IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)2i2c · 2022

Root cause Cloud provider outage

IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico componentGoogle Cloud · 2022

Root cause Software defect

IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1GitLab · 2021

Root cause Cloud provider outage

IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failingloveholidays · 2020

Root cause Capacity exhaustion

IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outageGrafana Labs · 2019

Root cause Cloud provider outage

IL-0039 New Kubernetes worker nodes could not join any cluster because a package URL hardcoded in kops nodeup brokeFREE NOW · 2019

Root cause Dependency failure

IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPAloveholidays · 2019

Root cause Cloud provider outage

IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod evictionOath · 2018

Root cause Software defect

IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker downSaltside · 2017

Root cause Capacity exhaustion