Incidents · Incident Library
Why are my Kubernetes pods stuck Pending with no available IP addresses?
Your pods are stuck because the cluster's network plugin cannot get an IP address for them. On EKS the AWS VPC CNI takes pod addresses from the VPC subnets and is limited by the node's network interfaces; on GKE each node reserves a block of the pod range in advance. When either runs out, new pods wait, and so do new nodes.
What the pod is waiting for
A pod gets its IP address from the cluster's network plugin, the CNI, when its network sandbox is created. If the plugin has no free address, the kubelet retries and the pod sits in ContainerCreating with an event that says the sandbox could not be created, or stays Pending because the scheduler already knows the node has no room for another pod IP. AWS documents the EKS form of this error as the CNI being unable to assign an IP address to the pod; the AWS re:Post knowledge base lists it under pods stuck in ContainerCreating.
Where the addresses come from depends on the platform. The AWS VPC CNI gives every pod its own address from the VPC subnet, through secondary addresses on the node's elastic network interfaces, so both the subnet and the instance type set a limit. VPC-native GKE clusters take pod addresses from a secondary range of the subnet and hand every node a fixed block of that range in advance.
The subnet ran out: Adevinta on EKS
Adevinta's platform team wrote up how an EKS cluster of about 10,000 pods used 10,000 VPC addresses, and how their first cluster ran out of IPv4 space in June 2024 during a migration. The alert that reached the on-call engineer was that some DaemonSets were failing; the pod status showed the aws-cni error about failing to assign an IP address.
They had three options on the table: replacing the plugin with Cilium, VPC CNI custom networking with a secondary CIDR, and IPv6. They chose custom networking because it was the quickest to put in place, rolled it out to the smaller clusters first and then to the whole fleet, and the migration finished without the outage they feared. Months later the shortage alerts came back when Karpenter started provisioning smaller instances, which carry fewer interfaces and addresses, and did not know about the secondary range; a --reserved-enis setting on Karpenter fixed the pod count per node. Their closing advice: for a large cluster started today, IPv6 is the cheaper and more future-proof choice.
The node's block ran out: loveholidays on GKE
loveholidays hit the limit from the other side. Their GKE cluster had a /16 pod range, which looks like room for 65,536 pods, yet it stopped growing at 256 nodes with new pods Pending and scale-ups failing. GKE allows 110 pods per node by default and gives each node a block holding at least twice that number, so each node took 256 addresses and 65,536 divided by 256 is 256 nodes.
The most effective fix was lowering the maximum pods per node on node pools that never run many pods, which cut IP consumption by about 30 percent. They also moved some pools to larger instances, which pack more pods per node at the cost of losing more pods when one node fails. Two commands helped them see it: describing the cluster-autoscaler-status ConfigMap in kube-system for the autoscaler's view, and reading a node's status.allocatable.pods for its maximum.
Nothing ran out, the plugin stopped: two more cases
On the UK Ministry of Justice Cloud Platform, six nodes stopped allocating pod addresses in July 2023 and new pods stayed in ContainerCreating for four and a half hours. The cause was a missing setting: comparing the live cluster's CNI configuration with a test cluster showed that prefix delegation was not enabled on live. Turning it on ended the incident. Their one follow-up action was a check that confirms address allocation works.
At CoW Protocol, a capacity expansion run through Infrastructure as Code also updated the VPC CNI plugin, because its version was not pinned. The resulting version mismatch meant new pods never got an address, updates and terminations stopped working, and settlements stopped for about three and a half hours until new node groups with a matching plugin version were set up. Their lesson was to pin component versions in IaC so an automation run cannot change the network plugin by accident.
How to tell which one you have
- Read the pod's events with kubectl describe. A sandbox error naming the CNI, or an event about no available IP, points at the plugin; a scheduling message about pod capacity points at the per-node limit.
- On EKS, read the aws-node logs on the node. AWS's troubleshooting guidance lists the plugin's log messages, and an insufficient-free-addresses error names the subnet. Check the subnet's free address count in the VPC console and the node's instance type against the number of interfaces and addresses it allows.
- On GKE, compare the pod range size with the number of nodes times the per-node block, and read the cluster autoscaler status ConfigMap for a scale-up that failed for lack of addresses.
- If addresses are plentiful and the failure started after a change, compare the CNI configuration and version with a cluster that works, as the Ministry of Justice and CoW Protocol teams did.
What the fixes have in common
Every public case above ended with more address space or fewer addresses per node: a secondary CIDR, prefix delegation, a lower maximum pods per node, larger instances, or IPv6. None of them was fixed by restarting pods. The two configuration cases were fixed by restoring a setting or a version, and both teams added a check or a pin so the same change could not slip through again.
Incident Library holds these five cases and others tagged with the symptom no-available-ip, each with a link to the original report, and its search accepts the symptom as a sentence. An AI assistant can read the same cases through the MCP server at https://incidentlib-mcp.benchlet.com/mcp, and a script through GET https://incidentlib-api.benchlet.com/search?q=pods pending no available ip address.
Frequently asked questions
Does restarting the node or the CNI pod free up addresses?
Only if the plugin itself was stuck. When the subnet or the node's block is exhausted, a restart changes nothing; the addresses are held by running pods or reserved by the node, and the fix is more space or fewer addresses per node.
Why does a /16 pod range fill up at 256 nodes on GKE?
Because GKE reserves a block of at least twice the maximum pods per node for every node, so at the default 110 pods per node each node takes 256 addresses, as loveholidays measured. Lowering the maximum pods per node on pools that run few pods shrinks the block.
Is IPv6 the answer on EKS?
Adevinta's team concluded that for large clusters started today IPv6 is the cheaper and more future-proof option, but chose custom networking with a secondary CIDR for an existing fleet because it was quicker and had fewer unknowns. The AWS documentation describes both.
Can the autoscaler add nodes to get more addresses?
Not when the pod range or subnet is the limit. In the loveholidays case node creation itself failed once the range was full, and the cluster autoscaler status ConfigMap recorded the failed scale-ups.