Cases2 cases

1 to 2 of 2

1. IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletion

   Neon2025

   Symptoms

   On 16 May 2025, in Neon's Kubernetes cluster in AWS us-east-1, new compute pods were scheduled but could not start and failed with IP allocation errors. Each Neon compute is a Postgres database running in a lightweight virtual machine in its own pod, so without new pods, idle customer databases could not be woken up.

   IP address exhaustionPods Pending

   Trigger

   On 16 May the Neon control plane's periodic job that terminates idle computes started failing. Compute creations continued, and the number of active computes rose from about 5k to about 8.1k within minutes. On 19 May, a follow-up change that set WARM_IP_TARGET back to 1 started the second round of failures.

   Root cause

   Capacity exhaustion

   Component

   AWS VPC CNIVPC subnet or pod address rangeCluster autoscaler

   Contributing factor

   Unreviewed defaultsNo capacity planningUnverified assumptionHarmful manual actionIneffective rollbackRetry storm

   Detection

   As cluster-autoscaler added nodes, the team saw IP allocation errors on new pods that were scheduled but could not start. The rate of successful pod starts and VPC subnet metrics were then used to follow the problem.

   Remediation

   • Set WARM_IP_TARGET=1 to release IPs held by nodes without free CPU or memory
   • Fixed the control plane's periodic job so idle computes were terminated again
   • Restarted the aws-node DaemonSet several times, each with a temporary improvement
   • Set WARM_IP_TARGET=0, MINIMUM_IP_TARGET=0 and WARM_ENI_TARGET=0, which let nodes allocate enough IPs and stabilised the cluster
   • On 19 May, set WARM_IP_TARGET back to 0 after setting it to 1 failed again
   • Enlarged the pool of pre-created warm compute pods to absorb failed pod starts

   Action items

   • Open a pull request to AWS CNI to improve how ipamd releases IPs in this situation
   • Follow up with AWS Support about the roughly 100 subnet IPs that were never allocated, for reasons Neon could not explain
   • Change the architecture to support horizontal scaling

   Lessons

   With default settings AWS CNI can keep many IPs on nodes that cannot use them, so the share of subnet IPs assigned to pods does not tell how close the subnet is to running out. WARM_IP_TARGET and WARM_ENI_TARGET follow different logic, a warm IP target of zero switches to the ENI logic, and IPs in cooldown count toward the target, which behaves badly under high pod churn. Changing these settings during an incident without knowing that logic prolonged the outage, and reverting a setting did not bring back the earlier healthy state.

   View case
2. IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failing

   loveholidays2020

   Symptoms

   A colleague wrote in Slack that his application was taking a long time to deploy. A deployment that normally finishes in minutes had been running for about an hour: about half of the new pods were running and serving traffic, and the other half were Pending.

   Pods PendingIP address exhaustionScale-up failure

   Trigger

   The cluster reaching 256 nodes, the most its /16 pod secondary range could hold at 256 addresses per node; every further node creation failed.

   Root cause

   Capacity exhaustion

   Component

   VPC subnet or pod address rangeCluster autoscalerNode pool

   Contributing factor

   Unreviewed defaultsUnknown limit

   Detection

   A colleague reported in Slack that a deployment was slow; the pending pods' events and the cluster autoscaler status ConfigMap showed the exhausted IP space.

   Remediation

   • lowered the maximum pods per node on selected node pools, cutting IP consumption by about 30%
   • moved some node pools to larger instance types

   Action items

   • Lower the maximum pods per node on node pools that do not run many pods
   • Use larger instance types in some node pools

   Lessons

   In a VPC-native GKE cluster the number of nodes, not the number of pods, uses up the pod range: by default each node reserves 256 addresses, so even a /16 is full at 256 nodes. Maximum pods per node and node size are the practical levers; larger nodes pack better but take more pods down when they fail.

   View case

By root cause1