Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

Node pool

노드 풀 · node-pool

36 cases · 2018 to 2026

Root cause Capacity exhaustion11 Configuration change6 Faulty deployment5 Software defect4 Cloud provider outage3 Certificate or credential expiry2 Dependency failure2 Operator error2 Breaking change in an upgrade1

Contributing factor Unreviewed defaults7 Automation amplification5 No alert on limit5 Single point of failure5

Symptom Service outage15 Scale-up failure9 Pods Pending8 DNS resolution failures5

Organisation UK Ministry of Justice6 Linode5 2i2c4 loveholidays3 Backpackr

Cases, by severity

IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hoursWealthy · 2025 · sev1
Root cause Breaking change in an upgrade
IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1
Root cause Configuration change

View on the graph

Monitoring blind spot4

Unrepresentative staging4

Excessive blast radius3

No configuration tests3

Overlapping failures3

Unverified assumption2

Limited debug access2

No resource requests or limits2

Ineffective rollback2

Deferred upgrades2

Change during peak hours1

Credential in configuration1

Ignored early signal1

Cache-masked failure1

Harmful manual action1

No capacity planning1

No safeguard on destructive change1

No staged rollout1

Pages not delivered1

Missed in review1

Shared credential1

Gap in automated tests1

Unpinned version1

High error rate4

IP address exhaustion4

Autoscaler misbehaviour3

Intermittent timeouts3

Container crash loop2

Credentials unavailable2

Deploys failing2

Image pull failures2

Intermittent connection failures2

Monitoring outage2

Sustained high node CPU2

Pods stuck in ContainerCreating2

Cluster or node provisioning stall1

Connection resets1

Control plane lag1

Control plane unresponsive1

Database overload1

GPU unavailable1

Node disk full1

Probe failures1

Resources deleted1

Stale or inconsistent API reads1

1

Microsoft Azure1

IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch

Buildkite · 2026

Root cause Capacity exhaustion

IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodesDigitalOcean · 2026

Root cause Capacity exhaustion

IL-0166 A compute host update removed the credential parser used to decode StackScripts, so new Linodes and every new LKE and LKE-E node failed to boot and autoscaling and node recycling failed in all regionsLinode · 2026

Root cause Faulty deployment

IL-0158 An expired credential caused authentication failures that stopped LKE and LKE-E from deploying clusters and nodes in all regions, which also blocked node recycling and cluster autoscalingLinode · 2026

Root cause Certificate or credential expiry

IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025

Root cause Capacity exhaustion

IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binariesMicrosoft · 2025

Root cause Software defect

IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouseRebelMouse · 2025

Root cause Capacity exhaustion

IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes2i2c · 2025

Root cause Capacity exhaustion

IL-0078 Enabling EKS secrets encryption with KMS through Terraform removed worker node IAM permissions and took down all ZenML Pro workspacesZenML · 2025

Root cause Software defect

IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025

Root cause Capacity exhaustion

IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went downUK Ministry of Justice · 2025

Root cause Faulty deployment

IL-0161 A faulty earlier API update made new LKE cluster provisioning in Toronto time out for three days, leaving clusters with no node pools or the wrong number of nodesLinode · 2024

Root cause Software defect

IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hoursCoW Protocol · 2024

Root cause Configuration change

IL-0021 GKE GPU nodes on Container-Optimized OS could not download NVIDIA drivers (403 Forbidden) and came up without working GPUsGoogle Cloud · 2024

Root cause Dependency failure

IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on deviceUK Ministry of Justice · 2023

Root cause Configuration change

IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023

Root cause Capacity exhaustion

IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreatingUK Ministry of Justice · 2023

Root cause Configuration change

IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods togetherUK Ministry of Justice · 2023

Root cause Configuration change

IL-0163 An API change that slipped through a gap in automated testing made LKE clusters resize to the wrong number of nodes and added billable worker nodes to customer accountsLinode · 2023

Root cause Faulty deployment

IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023

Root cause Capacity exhaustion

IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull imagesUK Ministry of Justice · 2023

Root cause Operator error

IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)2i2c · 2022

Root cause Cloud provider outage

IL-0164 An incremental Linode Kubernetes Engine production update that was missing changes from the previous release made cluster upgrades and node additions fail silentlyLinode · 2022

Root cause Faulty deployment

IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effectWebhook Relay · 2022

Root cause Certificate or credential expiry

IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1GitLab · 2021

Root cause Cloud provider outage

IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020

Root cause Capacity exhaustion

IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutesGrafana Labs · 2020

Root cause Software defect

IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020

Root cause Capacity exhaustion

IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failingloveholidays · 2020

Root cause Capacity exhaustion

IL-0149 Upgrades of AKS clusters on Virtual Machine Scale Sets failed with a 'platform image is not available' error after a service update left a backend service unhealthyMicrosoft Azure · 2019

Root cause Faulty deployment

IL-0039 New Kubernetes worker nodes could not join any cluster because a package URL hardcoded in kops nodeup brokeFREE NOW · 2019

Root cause Dependency failure

IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPAloveholidays · 2019

Root cause Cloud provider outage

IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx podsDevOps Hof · 2019

Root cause Operator error

IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a clusterTarget · 2018

Root cause Configuration change