IL-0007 Adevinta · 2024-10-22 (published)kubernetesawsnetworking
The AWS VPC CNI, the default network plugin on EKS, gives every pod its own IP address from the VPC, so a cluster of 10,000 pods uses 10,000 IPs. SCHIP is multi-tenant, and its pods reach tenant resources such as RDS databases and ElastiCache in other AWS accounts, first through VPC peering and later through a Transit Gateway. Because overlapping address ranges make those connections harder, the team has to maintain an internal IPAM. Adevinta had been monitoring its IP usage since early in its EKS work.
When the first cluster ran out of IPs in June, engineers had already spent several experimentation days, which the team gets every other Friday, testing three options: replacing the CNI with Cilium, VPC CNI custom networking, and IPv6 with the VPC CNI. They chose custom networking with a secondary CIDR because it was the easiest to put in place in time to finish the migration, though not the most elegant choice; Cilium and IPv6 each brought more unknowns.
Custom networking was rolled out in a controlled way to the smaller clusters from late August. In September available IPs were so low that the migration was paused; after the controlled rollout proved successful, custom networking went to the rest of the fleet, available IPs recovered, and the migration finished without the fleet-wide outage the team feared.
Months later, Karpenter began provisioning smaller instances, and the number of ENIs and IPs a node gets depends on its instance size. Karpenter was not aware of the secondary CIDR and custom networking, so IP shortage alerts returned. Adding the --reserved-enis parameter to the Karpenter deployment let Karpenter compute how many pods fit on a node, taking into account that the first ENI is used for custom networking.
One evening the platform team that runs SCHIP, Adevinta's internal runtime platform on Kubernetes, was paged because some DaemonSets were failing. Checking the pod status showed this error from the network plugin: name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
The CNI was saying plainly that no IP address was available for new pods. The affected cluster was small and little used, so the impact was small, but the page came in June, in the middle of a large migration of workloads to EKS-based clusters, and the same shortage could hit the bigger clusters.
In September the number of available IPs fell so low that the migration had to be paused. Months later, after the move to Karpenter started, pager alerts about a lack of IP addresses came back.
Root cause Capacity exhaustion
A pager alert that some DaemonSets were failing. The on-call engineer checked the pod status and found the aws-cni error about failing to assign an IP address.
For a platform team, the VPC CNI is not ready to scale in production without time spent on capacity planning, and every option has its pros and cons. For large clusters started today, IPv6 is the cheaper and more future-proof option. Do not assume that what works in small clusters will keep working when they grow.
IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletionNeon · 2025same root cause · shared AWS VPC CNI, VPC subnet or pod address range
IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation이언철 · 2024same root cause · shared Karpenter
IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failingloveholidays · 2020same root cause · shared VPC subnet or pod address range
IL-0015 linked by the editorIL-0019 linked by the editorhttps://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.