Cases7 cases

1 to 5 of 7

1. IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletion

   Neon2025

   Symptoms

   On 16 May 2025, in Neon's Kubernetes cluster in AWS us-east-1, new compute pods were scheduled but could not start and failed with IP allocation errors. Each Neon compute is a Postgres database running in a lightweight virtual machine in its own pod, so without new pods, idle customer databases could not be woken up.

   IP address exhaustionPods Pending

   Trigger

   On 16 May the Neon control plane's periodic job that terminates idle computes started failing. Compute creations continued, and the number of active computes rose from about 5k to about 8.1k within minutes. On 19 May, a follow-up change that set WARM_IP_TARGET back to 1 started the second round of failures.

   Root cause

   Capacity exhaustion

   Component

   AWS VPC CNIVPC subnet or pod address rangeCluster autoscaler

   Contributing factor

   Unreviewed defaultsNo capacity planningUnverified assumptionHarmful manual actionIneffective rollbackRetry storm

   Detection

   As cluster-autoscaler added nodes, the team saw IP allocation errors on new pods that were scheduled but could not start. The rate of successful pod starts and VPC subnet metrics were then used to follow the problem.

   Remediation

   • Set WARM_IP_TARGET=1 to release IPs held by nodes without free CPU or memory
   • Fixed the control plane's periodic job so idle computes were terminated again
   • Restarted the aws-node DaemonSet several times, each with a temporary improvement
   • Set WARM_IP_TARGET=0, MINIMUM_IP_TARGET=0 and WARM_ENI_TARGET=0, which let nodes allocate enough IPs and stabilised the cluster
   • On 19 May, set WARM_IP_TARGET back to 0 after setting it to 1 failed again
   • Enlarged the pool of pre-created warm compute pods to absorb failed pod starts

   Action items

   • Open a pull request to AWS CNI to improve how ipamd releases IPs in this situation
   • Follow up with AWS Support about the roughly 100 subnet IPs that were never allocated, for reasons Neon could not explain
   • Change the architecture to support horizontal scaling

   Lessons

   With default settings AWS CNI can keep many IPs on nodes that cannot use them, so the share of subnet IPs assigned to pods does not tell how close the subnet is to running out. WARM_IP_TARGET and WARM_ENI_TARGET follow different logic, a warm IP target of zero switches to the ENI logic, and IPs in cooldown count toward the target, which behaves badly under high pod churn. Changing these settings during an incident without knowing that logic prolonged the outage, and reverting a setting did not bring back the earlier healthy state.

   View case
2. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server

   Render2024

   Symptoms

   In January 2022 an etcd memory spike overloaded the Kubernetes control plane of one of Render's clusters in Frankfurt. At first only operations that need the control plane failed: new deploys and cron job runs.

   Control plane unresponsiveDeploys failingDNS resolution failuresHigh error rate

   Trigger

   A memory spike in etcd overloaded the cluster's control plane.

   Root cause

   Capacity exhaustion

   Component

   etcdCoreDNSKubernetes API server

   Contributing factor

   Unreviewed defaultsHarmful manual action

   Action items

   • Run CoreDNS on data plane nodes instead of control plane nodes
   • Run etcd on dedicated nodes

   Lessons

   CoreDNS can keep answering from its in-memory cache while the API server is down, but it cannot initialise without it. With CoreDNS on the control plane nodes, a reboot or restart during a control plane outage turned a failure of deploys and cron jobs into a failure of requests to services that were already running. The core problem was that the data plane was not isolated enough from the control plane: even without the reboot, the etcd memory spike alone could have forced CoreDNS to restart.

   View case
3. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation

   이언철2024

   Symptoms

   When traffic changed sharply, CoreDNS came under heavy load, and the cluster saw network problems and a sudden rise in latency. The horizontal pod autoscaler then added pods, database connections surged, and the impact spread to the whole service.

   High latencyIntermittent connection failuresAutoscaler misbehaviour

   Trigger

   Sharp changes in traffic loaded the small number of CoreDNS pods beyond what they could handle.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSKarpenterHorizontal pod autoscalerPod scheduling and placement

   Contributing factor

   Automation amplificationHarmful manual action

   Remediation

   • Set up the Cluster Proportional Autoscaler for CoreDNS with nodesPerReplica and coresPerReplica chosen for the instance types in use
   • Spread CoreDNS with soft topologySpreadConstraints (ScheduleAnyway) instead of anti-affinity so Karpenter could consolidate nodes
   • Added alert thresholds on CoreDNS error rate, panics, request duration and request rate
   • Deployed NodeLocal DNSCache so each node answers DNS from a local cache and forwards misses to the main CoreDNS

   Lessons

   A fixed, small number of CoreDNS pods cannot follow a cluster whose nodes and pods change quickly with traffic, and the damage spreads: DNS latency triggers autoscaling, which in turn floods the database with connections. Simply adding many CoreDNS replicas created a different problem, because with tainted nodes and a maxSkew: 1 spread constraint the pods piled onto a few nodes and blocked Karpenter consolidation. Scaling CoreDNS in proportion to the cluster and putting a DNS cache on every node handled both.

   View case
4. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day

   DFDS2024

   Symptoms

   At 10:01 UTC on 2024-03-13 a Kubernetes node entered NotReady state, with CPU requests at 78% and memory requests at 82%. An alert fired in Slack at 10:03 and was picked up at 10:05. An attempt to SSH into the node at 10:13 failed because it was unresponsive; it started responding again at 10:19.

   Node NotReadyContainer crash loopOOMKilledMonitoring outage

   Trigger

   Pods of a dev container that used Hangfire to trigger jobs failed with OOM and crash-looped on a node where a Grafana Agent pod was already using a lot of memory.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agentContainer memory limitApplication database

   Contributing factor

   Excessive blast radiusNo resource requests or limitsHarmful manual actionMonitoring blind spot

   Detection

   A Slack alert fired two minutes after the node went NotReady and was picked up two minutes later; the second node's NotReady at 15:03 also raised a Slack alert.

   Remediation

   • marked the EC2 instance unhealthy so a new node started
   • cordoned and force-drained the NotReady node and terminated its EC2 instance
   • removed the new LimitRange from the Grafana Agent namespace and restarted the Grafana Agent pod
   • scaled the dev container to 0 replicas, after which EC2 network traffic returned to normal
   • after the second NotReady, scaled the dev container to 0 replicas again and added a new node
   • emailed the team that owns the dev container, which confirmed it was working on a fix

   Action items

   • Move grafana-agent to the monitoring node group
   • Add alerts for when bandwidth across the cluster reaches a certain threshold
   • Investigate whether memory limits can be set on the grafana-agent Helm chart
   • Investigate grafana-agent memory usage
   • Ask Grafana support whether grafana-agent should use a PVC
   • Consider pushing node logs to Loki with limited retention
   • Create a wiki page with common metrics and log queries to check when investigating issues with Kubernetes nodes

   Lessons

   Dev workloads mixed with production let one crash-looping dev container take down nodes that production also used, and a monitoring agent outside its dedicated node group added to the memory pressure. Scaling the suspect container back up before its owners had fixed it brought the failure back on the replacement node. The team also noted that action items from earlier post-mortems need follow-up, and that knowledge of useful CloudWatch metrics had not been shared, which made the diagnosis harder.

   View case
5. IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memory

   UK Ministry of Justice2020

   Symptoms

   At 10:43 on 14 August 2020 two of the six ingress controller pods in the cluster were in a crash loop. Restarting those pods did not fix them.

   Container crash loop

   Root cause

   Capacity exhaustion

   Component

   Ingress controller

   Contributing factor

   Harmful manual actionSingle point of failure

   Detection

   Two of the six ingress controller pods were seen crash-looping at 10:43; the incident was declared at 11:01.

   Remediation

   • restarted the leader ingress controller pod so the other replicas could connect and get the latest nginx configuration

   View case

By root cause1