IL-0015 loveholidays · 2020-02-03 (published)kubernetesgcpnetworking
loveholidays uses Shared VPC to keep networking and compute in separate projects, and runs VPC-native GKE clusters, which take pod IP addresses from a secondary range of the subnet. The affected cluster had a dedicated subnet whose secondary range was a /16, which looks like room for 65,536 pods.
GKE allows 110 pods per node by default and pre-allocates addresses for them on every node, however few pods actually run there. The team expected that to cap the cluster at 595 nodes (65,536 / 110), but GKE also gives each node a block holding at least twice the maximum pod count, so that addresses are not reused too quickly as pods come and go. Each node therefore took 256 addresses, and 65,536 / 256 = 256 nodes, exactly where the cluster stopped growing.
The most effective fix was lowering the maximum pods per node on node pools that never run many pods, such as nodes for stateful applications, caches or security-sensitive workloads, which cut overall IP consumption by about 30%. The team also moved some node pools to larger instance types, which run more pods per node and pack them better, at the cost of more pods lost when one node fails. Expanding the subnet, which GCE supports by shortening the mask when no range collides, was considered and not done, because the documentation says nothing about secondary ranges. Two commands helped: kubectl describe -n kube-system configmap cluster-autoscaler-status for the autoscaler state, and kubectl get node NODE-NAME -ojson | jq .status.allocatable.pods for a node's maximum pods.
A colleague wrote in Slack that his application was taking a long time to deploy. A deployment that normally finishes in minutes had been running for about an hour: about half of the new pods were running and serving traffic, and the other half were Pending.
It did not look like the usual causes of a stuck deployment, such as a missing ConfigMap or Secret or a wrong nodeSelector. Describing one of the pending pods showed "Warning FailedScheduling 2m18s (x4 over 3m48s) default-scheduler 0/256 nodes are available: 1 node(s) were unschedulable, 173 Insufficient memory, 214 node(s) didn't match node selector, 87 Insufficient cpu." The GKE cluster autoscaler, which normally adds nodes in this situation, was getting nowhere, and the round power-of-two number of 256 nodes looked suspicious.
The cluster autoscaler status, read with kubectl describe -n kube-system configmap cluster-autoscaler-status, showed repeated failures: "Warning ScaleUpFailed 11m (x14 over 36m) cluster-autoscaler (combined from similar events): Failed adding 3 nodes to group ... Instance 'INSTANCE' creation failed: IP space of 'projects/PROJECT/regions/OUR_REGION/subnetworks/SUBNETWORK' is exhausted." That was puzzling, because the cluster's pod range was a /16 with 65,536 addresses and the cluster ran thousands of pods, far from that number.
Root cause Capacity exhaustion
A colleague reported in Slack that a deployment was slow; the pending pods' events and the cluster autoscaler status ConfigMap showed the exhausted IP space.
In a VPC-native GKE cluster the number of nodes, not the number of pods, uses up the pod range: by default each node reserves 256 addresses, so even a /16 is full at 256 nodes. Maximum pods per node and node size are the practical levers; larger nodes pack better but take more pods down when they fail.
IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletionNeon · 2025same root cause · shared Cluster autoscaler, VPC subnet or pod address range
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared Node pool
IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodesDigitalOcean · 2026same root cause · shared Node pool
IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared Node pool
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Node pool
IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes2i2c · 2025same root cause · shared Node pool
IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared Node pool
IL-0007 The AWS VPC CNI could not assign IP addresses to pods as an EKS cluster ran out of IPv4 spaceAdevinta · 2024same root cause · shared VPC subnet or pod address range
IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023same root cause · shared Node pool
IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023same root cause · shared Node pool
IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020same root cause · shared Node pool
IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020same root cause · shared Node pool
IL-0017 linked by the editor