EKS "failed to assign an IP address to container" and awscni_no_available_ip_addresses: start-up race or IP exhaustion?

What we saw

!The failed IP assignment on node-a at 12:32:40.926 UTC on a zoomed time axis: the line "DataStore has no available IP/Prefix addresses", eni-2's three IPs added to the datastore 10 ms later at 12:32:40.936, and the next request given an IP at 12:32:41.406. *The one failure behind the counter, on 2026-11-07: 10 ms before the second ENI's IPs reached the datastore, 0.48 s before the next request got an IP.*

If a pod on your EKS node got a FailedCreatePodSandBox event with add cmd: failed to assign an IP address to container, ipamd logged DataStore has no available IP/Prefix addresses, and the metric awscni_no_available_ip_addresses is above zero, the node is not necessarily out of pod IPs, and the pod is not necessarily still stuck in ContainerCreating. On ours the counter stayed at 1 for eight days because of one failed request in the node's first seconds. With WARM_ENI_TARGET=1 the second ENI is attached only after the first pod takes an IP, and the fourth pod that needed an IP (host-network pods such as aws-node and kube-proxy take none) arrived 10 ms before that ENI's IPs were usable. The next request got an IP 0.48 s later. Below: the node log of that sequence, the VPC CNI source lines behind it, and five read-only checks with the values that separate the race from a real shortage. The cluster is a 9-node EKS cluster (Kubernetes 1.32, VPC CNI add-on v1.19.0-eksbuild.1 with no configuration values); the node is node-a, a t8i.small; all times are UTC.

A full run of kubegap v0.3.0, an open-source audit of Kubernetes cluster operations that I maintain, on 2026-11-14 at 16:51 listed this among the problems after its top five (translated): "A system node that could not hand out a pod IP. On node-a (t8i.small) ipamd answered 'no available IP' once. IP use is now 6 of 9 and max-pods is 11, so pods can get stuck in ContainerCreating when many arrive at once. Severity high." The finding behind it:

vpccni.no-available-ip | target node-a | fail | high
  node node-a (t8i.small, pool system): ipamd answered 'no available IP' 1 times since it started;
  pod IPs 6/9 (0.667), ENI 3/3, max-pods 11

The rule fired when the counter was above 0. Its explanation said any non-zero value "means a pod on this node has already stalled in ContainerCreating".

The counter on the node on 2026-11-15 at 13:46:23, read with curl 127.0.0.1:61678/metrics from inside the node (the apiserver pod proxy to port 61678 did not answer within 2 minutes because of the node security group):

awscni_no_available_ip_addresses 1
awscni_assigned_ip_addresses 6
awscni_total_ip_addresses 9
awscni_eni_allocated 3
awscni_eni_max 3
awscni_total_ipv4_prefixes 0
awscni_add_ip_req_count 16
awscni_force_removed_ips 0

It had also been 1 in the snapshot of the kubegap run. kubegap read the metrics of 7 of the 9 nodes; the other 6 it read had 0.

Nothing else showed a problem:

- kubectl get events -A --field-selector reason=FailedCreatePodSandBox at 13:45:20 returned nothing. Events are kept for about an hour, so this says nothing about the day the counter moved.

- The cluster's Prometheus does not scrape awscni_*: count(awscni_no_available_ip_addresses) returned no series.

- The ipamd log of the last 24 hours, warn and above, had no line.

- At 13:42:58 node-a ran 11 pods: 9 running, 2 completed CI jobs. Of the 9 running, 3 use host networking (aws-node, kube-proxy, node-exporter) and 6 have a pod IP (argocd-redis, ebs-csi-node, kafka-exporter, grafana, promtail and an app's PostgreSQL).

The node was created on 2026-11-07 at 12:32:25 by Karpenter (on-demand, system NodePool). The only "failed to assign" line in its plugin.log is from that minute:

2026-11-07T12:32:40.927Z error routed-eni-cni-plugin/cni.go:128       Failed to assign an IP address to container <sandbox 4>

Why `failed to assign an IP address to container`: the fourth pod came before the second ENI's IPs

The node's capacity: 3 IPs per ENI

A t8i.small has 3 network interfaces with 4 IPv4 addresses each (EC2 DescribeInstanceTypes). Each ENI keeps one address for itself, so pods get 3 × (4 − 1) = 9 IPs, 3 per ENI. AWS's max-pods formula adds 2 for host-network pods: 9 + 2 = 11, the allocatable pods of node-a. The VPC CNI on this cluster runs with the defaults: WARM_ENI_TARGET=1, WARM_PREFIX_TARGET=1, ENABLE_PREFIX_DELEGATION=false, no WARM_IP_TARGET, no MINIMUM_IP_TARGET, ipamd log at DEBUG in /host/var/log/aws-routed-eni/ipamd.log inside the aws-node container, which is /var/log/aws-routed-eni/ipamd.log on the node.

`DataStore has no available IP/Prefix addresses`: the first 24 seconds of ipamd's log

The rotated ipamd log on node-a still held the node's start. The file is JSON lines ({"level":"info","ts":"…","caller":"…","msg":"…"}); the excerpts below show time, level, caller and message. The caller field does not match the v1.19.0 source lines cited in the next section (the error shows data_store.go:607, the source has it at data_store.go:734; the pool manager line shows runtime/asm_amd64.s), so grep for the message text, not the caller. ipamd started 6 s after the node object was created, filled the primary ENI with 3 secondary IPs and logged its pool settings:

2026-11-07T12:32:31.588Z info  aws-k8s-agent/main.go:42               Starting L-IPAMD   ...
2026-11-07T12:32:32.853Z info  ipamd/ipamd.go:931                     Trying to allocate 3 IP addresses on ENI eni-1
2026-11-07T12:32:33.297Z info  ipamd/ipamd.go:1121                    Added ENI(eni-1)'s IP/Prefix 192.0.2.11/32 to datastore
2026-11-07T12:32:33.297Z info  ipamd/ipamd.go:1121                    Added ENI(eni-1)'s IP/Prefix 192.0.2.12/32 to datastore
2026-11-07T12:32:33.297Z info  ipamd/ipamd.go:1121                    Added ENI(eni-1)'s IP/Prefix 192.0.2.13/32 to datastore
2026-11-07T12:32:33.297Z info  runtime/asm_amd64.s:1695               IP pool manager - max pods: 11, warm IP target: 0, warm prefix target: 1, warm ENI target: 1, minimum IP target: 0

Two pods took two of the three IPs. 1.7 s after the first one, ipamd began creating a second ENI:

2026-11-07T12:32:34.052Z info  datastore/data_store.go:714            assignPodIPAddressUnsafe: Assign IP 192.0.2.11 to sandbox aws-cni/<sandbox 1>/eth0
2026-11-07T12:32:34.203Z info  datastore/data_store.go:714            assignPodIPAddressUnsafe: Assign IP 192.0.2.12 to sandbox aws-cni/<sandbox 2>/eth0
2026-11-07T12:32:35.798Z info  awsutils/awsutils.go:795               Trying to allocate 3 IP addresses on new ENI
2026-11-07T12:32:36.504Z info  awsutils/awsutils.go:941               Created a new ENI: eni-2
2026-11-07T12:32:37.800Z info  ipamd/ipamd.go:865                     Successfully created and attached a new ENI eni-2 to instance

The third pod took the last IP of eni-1. The fourth request found none; 10 ms later eni-2's IPs were in the datastore:

2026-11-07T12:32:40.738Z info  datastore/data_store.go:714            assignPodIPAddressUnsafe: Assign IP 192.0.2.13 to sandbox aws-cni/<sandbox 3>/eth0
2026-11-07T12:32:40.926Z error datastore/data_store.go:607            DataStore has no available IP/Prefix addresses
2026-11-07T12:32:40.927Z info  rpc/rpc.pb.go:863                      Send AddNetworkReply: IPv4Addr: , IPv6Addr: , DeviceNumber: -1, err: AssignPodIPv4Address: no available IP/Prefix addresses
2026-11-07T12:32:40.936Z info  ipamd/ipamd.go:886                     Found ENIs having 4 secondary IPs and 0 Prefixes
2026-11-07T12:32:40.936Z info  ipamd/ipamd.go:1121                    Added ENI(eni-2)'s IP/Prefix 192.0.2.15/32 to datastore
2026-11-07T12:32:40.936Z info  ipamd/ipamd.go:1121                    Added ENI(eni-2)'s IP/Prefix 192.0.2.16/32 to datastore
2026-11-07T12:32:40.936Z info  ipamd/ipamd.go:1121                    Added ENI(eni-2)'s IP/Prefix 192.0.2.17/32 to datastore

The failed sandbox was torn down (datastore: unknown pod at 12:32:40.947), a new one was created at 12:32:41.405, and it got the first IP of eni-2 at 12:32:41.406, 0.479 s after the failure:

2026-11-07T12:32:41.406Z info  datastore/data_store.go:714            assignPodIPAddressUnsafe: Assign IP 192.0.2.15 to sandbox aws-cni/<sandbox 5>/eth0

From 12:32:45.937 to 12:32:51.031 a third ENI, eni-3, was created, attached and its 3 IPs added. From 12:32:51 the node had 9 IPs, and at 12:33:24 the last pod of the start-up got an IP.

!Measured timeline of node-a from ipamd start, seconds 0 to 24, one lane per ENI and one lane of pod IP requests: eni-1's 3 IPs at +1.7 s, pods 1 to 3 taking them, eni-2 created and attached from +4.2 s to +6.2 s, the failed fourth request at +9.3 s, eni-2's IPs added 10 ms later, the next request served at +9.8 s, eni-3's IPs at +19.4 s, the sixth request at +22.6 s. *node-a on 2026-11-07, seconds since Starting L-IPAMD at 12:32:31.588 UTC. The gap is between the attach of eni-2 and its IPs reaching the datastore.*

Why the second ENI was being attached only then

Each step below is in the amazon-vpc-cni-k8s source at tag v1.19.0:

- At start, nodeInit calls increaseDatastorePool once. One call either fills an existing ENI or attaches a new one, never both, so a new node starts with the primary ENI's 3 secondary IPs and no second ENI (ipamd.go:565–567, :810–833).

- With WARM_ENI_TARGET=1 and 3 IPs per ENI, the pool counts as too low when fewer than 3 IPs are free (ipamd.go:2178–2179). With 3 free IPs at start, 3 < 3 is false. The pool becomes low when the first pod takes an IP.

- The pool manager checks every 5 s (ipamd.go:58, :656–665) and attaches one ENI per step. Our log matches: first pod at 12:32:34.052, new ENI started at 12:32:35.798.

- After the attach, ipamd waits until the ENI and its secondary IPs appear in instance metadata (waitForENIAndIPsAttached, awsutils.go:1742), sets up the ENI's network and only then adds the IPs to the datastore (ipamd.go:844–893). On node-a the IPs reached the datastore 3.1 s after the attach (12:32:37.800 → 12:32:40.936). The node's own latency metric for waitForENIAndIPsAttached totals 140 ms for both ENIs, so most of the 3.1 s came after the metadata wait; we did not trace which step took it.

- The gRPC handler serves each CNI ADD from the datastore without waiting for the pool manager (rpc_handler.go:174). AssignPodIPv4Address walks every ENI, finds no free IP, increments the counter, logs DataStore has no available IP/Prefix addresses and returns the error (data_store.go:680–735). The plugin then returns add cmd: failed to assign an IP address to container to the runtime (cni.go:185), and kubelet records FailedCreatePodSandBox and retries.

AWS's best-practices page describes the same order: "The IPAMD allocates additional ENI when a single Pod is scheduled and assigned a secondary IP address from the primary ENI." The repository's docs/eni-and-ip-target.md says "Creating and attaching a new ENI to a node can take up to 10 seconds." In issue #2669, titled "DataStore has no available IP/Prefix addresses", a maintainer wrote that the log line is "not out of the ordinary if the node is scaling up and attaching new ENIs"; in issue #3255 a maintainer confirmed that a temporary FailedCreatePodSandBox is expected while the pool is momentarily empty.

So on a node with 3 IPs per ENI that receives more than 3 IP-using pods in its first seconds, one request can find the pool empty. Our conclusion (an inference from the log and the source): the single count is the fourth IP request on the node, made 9.3 s after ipamd started while the second ENI was 10 ms from usable, and it did not recur.

We did not see the pod event for it: events had expired long before we looked. The event text above is what the plugin source returns, not something we captured.

What `awscni_no_available_ip_addresses` counts

awscni_no_available_ip_addresses is a Prometheus counter that ipamd increments in AssignPodIPv4Address just before it logs the error (data_store.go:733–735, prometheusmetrics.go:162–167). It is reset only when the aws-node container restarts, and it counts failed assignment calls, not pods: a pod whose IP request fails on two sandbox attempts adds 2. A value of 1 is history since aws-node started, not the current state. On node-a it was 1 at the kubegap snapshot, 172.3 hours after ipamd started, and still 1 the next day. A maintainer in issue #2649 lists five reasons the counter moves, among them "CNI waiting for new ENIs to be attached", and adds "we do not really delineate these cases today".

What to check in your cluster now

All steps are read-only. Steps 3 and 4 run on the node (SSH or Session Manager), because the apiserver pod proxy to port 61678 usually times out when the node security group does not allow it; step 4 also needs root (sudo; AWS-RunShellScript already runs as root), because the files in /var/log/aws-routed-eni/ are mode 0600, owned by root. Replace <node>, <type> and, in step 4, <YYYY-MM-DDTHH:MM> (the minute of a failure from the first grep) with yours.

1. How many pod IPs can the node hold, and how old is it?

kubectl get node <node> -o jsonpath='{.metadata.labels.node\.kubernetes\.io/instance-type}{" created="}{.metadata.creationTimestamp}{" max-pods="}{.status.allocatable.pods}{"\n"}'
aws ec2 describe-instance-types --instance-types <type> --query 'InstanceTypes[].[MaximumNetworkInterfaces,NetworkInfo.Ipv4AddressesPerInterface]' --output text

Pod IPs = ENIs × (IPv4 per ENI − 1). Ours: t8i.small, 3 × (4 − 1) = 9, max-pods 11, created 8 days earlier. The smaller the IPs-per-ENI number, the fewer pods it takes to empty the first ENI.

2. Which warm settings does the VPC CNI run with?

kubectl -n kube-system get ds aws-node -o jsonpath='{range .spec.template.spec.containers[?(@.name=="aws-node")].env[*]}{.name}={.value}{"\n"}{end}' | grep -E 'WARM_|MINIMUM_IP|PREFIX_DELEGATION|LOG_FILE|LOGLEVEL'

Same setup as ours: WARM_ENI_TARGET=1, ENABLE_PREFIX_DELEGATION=false, no WARM_IP_TARGET or MINIMUM_IP_TARGET; AWS_VPC_K8S_CNI_LOG_FILE tells step 4 where the ipamd log is (ours /host/var/log/aws-routed-eni/ipamd.log, read on the node without /host), and AWS_VPC_K8S_CNI_LOGLEVEL=DEBUG makes it rotate within days. With these, the second ENI is attached only after the first pod takes an IP from the first one, so on a node with 3 IPs per ENI the fourth IP-using pod of the first seconds can find the pool empty.

3. What do the counter and the pool say now? (on the node)

curl -s http://127.0.0.1:61678/metrics | grep -E '^awscni_(no_available_ip_addresses|assigned_ip_addresses|total_ip_addresses|eni_allocated|eni_max) '

Same problem as ours (history, not a shortage) when the counter is small and does not grow when you read it again later, assigned is below total, and eni_allocated is below eni_max or total − assigned covers the pods still to come. Ours: counter 1 (and 1 about 21 hours earlier, in the kubegap snapshot), assigned 6 of 9, ENI 3 of 3.

A real shortage looks different: the counter keeps rising, assigned equals total, and kubectl get events -A --field-selector reason=FailedCreatePodSandBox shows failed to assign an IP address to container for pods that stay in ContainerCreating; eni_allocated then equals eni_max when the node is full, or stays below it when the subnet has no free address (check the subnet's AvailableIpAddressCount).

4. When did each failure happen? (on the node)

grep -h 'Failed to assign an IP address' /var/log/aws-routed-eni/plugin.log*
for f in /var/log/aws-routed-eni/ipamd*.log; do grep -h 'no available IP/Prefix' "$f"; done
zcat /var/log/aws-routed-eni/ipamd-*.log.gz 2>/dev/null | grep -h 'no available IP/Prefix'
zcat -f /var/log/aws-routed-eni/ipamd-*.log.gz /var/log/aws-routed-eni/ipamd.log | grep 'Starting L-IPAMD'   # every start; the counter counts from the last one

Then read the minute around each failure for Added ENI(…)'s IP/Prefix … to datastore:

zcat -f /var/log/aws-routed-eni/ipamd-*.log.gz /var/log/aws-routed-eni/ipamd.log | grep '"ts":"<YYYY-MM-DDTHH:MM>' | grep -v '"level":"debug"'

Start-up race when the failure is within seconds of Starting L-IPAMD (or of the node joining), the IPs of a newly attached ENI are added to the datastore milliseconds to seconds after it, and the next CNI ADD gets an IP. Ours: ipamd started 12:32:31.588, failure 12:32:40.926, eni-2's IPs added 12:32:40.936, the next ADD got an IP at 12:32:41.406. The log rotates by size; node-a logs at DEBUG and its log rotated on 2026-11-14 at 00:15, 6.5 days after the node started, so when we read it on 11-15 the start was only in ipamd-2026-11-14T00-15-03.555.log.gz.

5. Can this node ever run out in steady state?

kubectl get pods -A --field-selector spec.nodeName=<node>,status.phase=Running -o json | jq '[.items[] | select(.spec.hostNetwork == true)] | length'

max-pods minus host-network pods is the most pods that can ever ask for an IP. Ours: 11 − 3 = 8 against 9 IPs, so the node cannot run out once its three ENIs are attached, apart from IPs held in the 30-second cooldown after a pod is deleted.

!Two columns, start-up race and real shortage, comparing five signals: the counter (small and flat vs rising), assigned vs total IPs, allocated vs maximum ENIs, the time of the failure relative to ipamd start, and max-pods minus host-network pods against pod IPs. node-a's values are marked: counter 1, 6 of 9 IPs, 3 of 3 ENIs, failure 9.3 s after ipamd start, 8 against 9. *The readings from steps 3 to 5 that separate the two cases, with node-a's values: counter and IPs read on 2026-11-15, failure timing from its 2026-11-07 log.*

If you do not want to repeat step 3, step 5 and the last-24-hours part of step 4 on every node by hand, kubegap does it: it reads the counter, the ipamd log of the last 24 hours and the aws-node start time from each node through a read-only SSM document, fails only on a "no available IP" line more than 120 s after an ipamd start, and reports a count older than that log as vpccni.no-available-ip-unexplained with the max-pods check of step 5. As of 2026-11-15 no release has it: v0.3.0 (released 2026-11-14), whose report led to this article, fails on any non-zero counter, so install from main at commit 0adb216 or later (pull request #210, rubric 0.1.96).

What we changed, and what we have not

Nothing on the cluster

No VPC CNI setting was changed. The evidence shows no shortage: one failed request 9.3 s after ipamd started, the next request served 0.48 s later, and no failure in the 8 days after. A VPC CNI setting applies to every node in the cluster, and I would not change all nine nodes to remove one retried sandbox on one of them.

The audit rule

The rule that raised it was wrong for this case, so we fixed the rule (kubegap issue #209, opened 2026-11-15 14:04:26; pull request #210, merged 14:16:16 as commit 0adb216). It uses only what kubegap already collects; the node-side SSM document is unchanged.

- vpccni.no-available-ip (high) now fails only on a "no available IP" line in the last 24 hours of ipamd log that comes more than 120 s after an ipamd start. Lines within 120 s of a start are the start-up race and pass.

- A counter value that the 24-hour log cannot explain goes to a new check, vpccni.no-available-ip-unexplained (warn, low). It says when ipamd started and whether max-pods minus host-network pods fits the IP capacity.

- Why 120 s: the race came 9.3 s after ipamd started on node-a. The pool manager runs every 5 s and waits for a new ENI's IPs in instance metadata with a backoff that starts at 100 ms, doubles and adds up to 15 % jitter; through the tenth try the sleeps add up to at most 58.8 s, so 120 s covers one tick and that wait with room for slow EC2 calls.

Re-evaluated on the same snapshot with the new code, without collecting anything again (14:16:29), condensed from the before/after comparison:

vpccni.no-available-ip             node-a   fail/high  ->  pass
vpccni.no-available-ip-unexplained node-a   (new)      ->  warn/low
  ipamd answered 'no available IP' 1 times since it started at 2026-11-07T12:32:31Z
  (172.3 h before the snapshot) ... Pods that can need an IP:
  max-pods 11 - 3 hostNetwork = 8, against 9 pod IPs: fits
vpccni.prefix-delegation-off       cluster  fail       ->  warn
vpccni.node-ip-exhaustion          node-a   warn/high  (unchanged: 6/9 IPs in use, ENI 3/3)

!The old and new `vpccni.no-available-ip` rule as decision paths. Old: counter above 0, fail high. New: a "no available IP" line in the last 24 h of ipamd log more than 120 s after an ipamd start, fail high; within 120 s of a start, pass; a count older than the log, warn low with the max-pods fit. node-a's path is highlighted: count older than the log, 8 against 9, warn low. *The kubegap rule before and after pull request #210, with node-a's path on the 2026-11-14 snapshot.*

Not done

- Prefix delegation. From the source, it would remove the first-seconds window: the primary ENI gets a /28 at nodeInit. It would also need a max-pods change on this node type to use the extra addresses. Not tested.

- `MINIMUM_IP_TARGET`. From the source, it would not remove the window by itself: nodeInit still does one pool step, and further ENIs come from the 5-second loop. AWS describes it as a way "to front-load the assignment of multiple ENIs at instance launch"; we have not tested what it does on a t8i.small in its first seconds.

- A different instance type for the system pool. Not done, for the same reason as the settings: there is no shortage to fix.

- An alert on the counter. The cluster's Prometheus does not scrape awscni_* today. If we add it, I would alert on increase(awscni_no_available_ip_addresses[10m]) > 0 together with free IPs (awscni_total_ip_addresses - awscni_assigned_ip_addresses) near zero, not on the raw value being above 0.

Where else this happened

The Incident Library, our searchable collection of public post-incident reports on benchlet.com, has no case of a one-off failure on a new node or a false IP-exhaustion alarm. The cases under the same symptom and component are real exhaustion or real allocation outages, which is what a lasting failure looks like:

- Neon, 2025-05-16: on VPC CNI v1.18.6 with the defaults (WARM_ENI_TARGET=1, WARM_IP_TARGET unset), nodes full on CPU and memory kept about 3.4k IPs no pod could use, the subnets ran out, and setting WARM_IP_TARGET=1 then failed pod starts for 30 s after each pod deletion.

- UK Ministry of Justice, 2023-07-21: new pods on six nodes stayed in ContainerCreating because the VPC CNI allocated no IPs; after about three hours the team found the live cluster lacked the prefix delegation setting its test cluster had, and the incident took 4 h 27 min.

- Adevinta, 2024-10-22: an EKS cluster ran out of IPv4 space; the on-call engineer found the aws-cni error about failing to assign an IP address, and the fix was custom networking on a secondary CIDR.

The other two cases under the same filter are CoW Protocol (an unpinned VPC CNI update during a capacity change) and a second UK Ministry of Justice incident (pods in ContainerCreating with the same error during a node drain): No available IP + VPC CNI, 5 cases.

Timeline

Frequently asked questions