IL-0006 Neon · 2025-05-16kubernetesawsnetworking
Neon's us-east-1 cluster usually peaks at about 6,000 running computes, starting about 500 pods a minute and terminating idle ones at a similar rate. As one of Neon's first regions, its three subnets were /20 blocks, 12,273 usable addresses in total, half the size used in other regions, on the assumption from load tests that a cluster would not go beyond 10k computes. Growth had been faster than expected. The cluster ran AWS VPC CNI v1.18.6 with the default settings: WARM_ENI_TARGET=1 and WARM_IP_TARGET unset. Nodes were m6id.metal instances with 49 pod IPs per ENI.
With these defaults, ipamd, the AWS CNI daemon on each node, keeps at least one ENI's worth of spare IPs and can only release IPs when a whole ENI has no pod on it. Pods are spread randomly over ENIs (ipamd keeps ENIs in a Go map, whose iteration order is random), and with a steady flow of new pods nearly every ENI keeps at least one pod, so IPs once allocated to a node are never returned. When the idle-termination job failed and computes jumped to about 8.1k, cluster-autoscaler added nodes, while older nodes that were full on CPU and memory held about 3.4k allocated IPs that no pod could use. The subnets ran out.
To free those IPs the team set WARM_IP_TARGET=1, having misread the documentation to mean that the unset default was 5. Thousands of IPs were released and pods started until about 10k concurrent pods, where limits known from load testing, such as kube-proxy sync latency, slowed starts. After the control-plane job was stabilised and about 7k idle computes were terminated, the rate of successful pod starts stayed far below normal. Neon later found the reason in the AWS CNI source: an IP is held in a 30-second cooldown after a pod is removed, and IPs in cooldown count toward the warm target, so WARM_IP_TARGET=1 can block all pod starts on a node for 30 seconds after each pod deletion. The control plane retried compute creation after its 2-minute timeout, so many pods that had started were deleted before the rest of setup finished, each deletion restarting the 30-second wait.
As a last attempt the team set WARM_IP_TARGET=0, MINIMUM_IP_TARGET=0 and WARM_ENI_TARGET=0. A warm IP target of zero is the same as not setting it, so ipamd went back to ENI-based logic, stopped returning IPs and allocated as many IPs as fit on an ENI at a time. Allocated IPs per node rose from about 75 to about 250, pod starts returned to normal, and the cluster stayed stable over the weekend.
On 19 May, unsure whether zero was a defined setting and suspecting the recovery on Friday had been a coincidence, the team set WARM_IP_TARGET=1 again, and pod starts broke the same way. Reverting to 0 did not recover the cluster: with all three targets at zero, ipamd only allocates more IPs when every IP on the node is assigned and no pod has been removed in the last 30 seconds, which never happened under constant pod churn. The team hid the errors from users by enlarging the pool of pre-created warm compute pods. Hours later the errors stopped when the control plane, slowed by an expensive query to its Postgres database, did not stop any pods for 86 seconds; with no IPs in cooldown, ipamd could allocate IPs across the cluster.
On 16 May 2025, in Neon's Kubernetes cluster in AWS us-east-1, new compute pods were scheduled but could not start and failed with IP allocation errors. Each Neon compute is a Postgres database running in a lightweight virtual machine in its own pod, so without new pods, idle customer databases could not be woken up.
The numbers were confusing at the time. About 8.8k IP addresses (71%) were assigned to pods, yet about 12,200 of the 12,273 addresses in the cluster's three VPC subnets (99%) were allocated to nodes. The subnets seemed to have unassigned IPs to spare, but the new nodes that cluster-autoscaler added could not get enough IPs to match their free CPU and memory. The cluster had been tested for 10k computes, and it ran out of IPs at about 8k.
After WARM_IP_TARGET was set to 1, thousands of IPs went back to the subnets and pods started again. But once the control plane resumed terminating idle computes, the rate of successful pod starts stayed far below normal. Most new pods failed on IP assignment even though VPC subnet metrics showed thousands of unallocated addresses. The AWS CNI documentation warns that WARM_IP_TARGET can cause EC2 API rate limiting, but that did not seem likely at 1-2 pod starts per node per minute. Each restart of the aws-node DaemonSet, which reinitialises ipamd, brought a temporary improvement.
On 19 May, setting WARM_IP_TARGET=1 again made IP assignment fail immediately, this time from a stable state. Changing it back to 0 did not stop the errors: about 10% of pods kept failing to start for hours.
Root cause Capacity exhaustion
As cluster-autoscaler added nodes, the team saw IP allocation errors on new pods that were scheduled but could not start. The rate of successful pod starts and VPC subnet metrics were then used to follow the problem.
With default settings AWS CNI can keep many IPs on nodes that cannot use them, so the share of subnet IPs assigned to pods does not tell how close the subnet is to running out. WARM_IP_TARGET and WARM_ENI_TARGET follow different logic, a warm IP target of zero switches to the ENI logic, and IPs in cooldown count toward the target, which behaves badly under high pod churn. Changing these settings during an incident without knowing that logic prolonged the outage, and reverting a setting did not bring back the earlier healthy state.
IL-0007 The AWS VPC CNI could not assign IP addresses to pods as an EKS cluster ran out of IPv4 spaceAdevinta · 2024same root cause · shared AWS VPC CNI, VPC subnet or pod address range
IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failingloveholidays · 2020same root cause · shared Cluster autoscaler, VPC subnet or pod address range
IL-0007 linked by the editorIL-0015 linked by the editorIL-0017 linked by the editorhttps://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.