Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

AWS VPC CNI

AWS VPC CNI · vpc-cni

9 cases · 2019 to 2025

Root cause Configuration change4 Capacity exhaustion2 DNS or network fault2 Software defect1

Contributing factor Unreviewed defaults4 Unverified assumption3 No configuration tests3 No capacity planning2

Symptom IP address exhaustion5 Pods Pending3 High latency2 Intermittent connection failures2

Organisation DFDS2 UK Ministry of Justice2 Adevinta1 CoW Protocol1 MindTickle

Cases, by severity

IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletionNeon · 2025
Root cause Capacity exhaustion
IL-0007 The AWS VPC CNI could not assign IP addresses to pods as an EKS cluster ran out of IPv4 spaceAdevinta · 2024
Root cause Capacity exhaustion
IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours

View on the graph

Overlapping failures2

Automation amplification1

Unpatched known bug1

Harmful manual action1

Monitoring blind spot1

No alert on limit1

No health probes1

No staged rollout1

Ineffective rollback1

Missed in review1

Unpinned version1

Intermittent timeouts2

Pods stuck in ContainerCreating2

Service outage2

High error rate1

Image pull failures1

Node disk full1

1

CoW Protocol · 2024

Root cause Configuration change

IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on deviceUK Ministry of Justice · 2023

Root cause Configuration change

IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreatingUK Ministry of Justice · 2023

Root cause Configuration change

IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022

Root cause Software defect

IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503DFDS · 2021

Root cause Configuration change

IL-0056 Calls from EKS pods to services behind an NLB in another VPC failed intermittently because the AWS VPC CNI did source NAT on the nodeMindTickle · 2020

Root cause DNS or network fault

IL-0057 Higher p99 latency after moving a reverse proxy to EKS, from outgoing connections delayed by a SNAT conntrack racePrezi · 2019

Root cause DNS or network fault