Cases51 cases

11 to 15 of 51

1. IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse

   RebelMouse2025

   Symptoms

   At 13:42 UTC on 19 May 2025, RebelMouse's monitoring tools reported a higher error rate and degraded performance across services. A routine reroll of the EKS nodes had started 27 minutes earlier, at 13:15 UTC.

   DNS resolution failuresHigh latencyHigh error rateService outage

   Trigger

   At 13:15 UTC RebelMouse started its regular reroll of EKS nodes to apply security patches. The planned, automatic termination of an EC2 instance terminated one of the CoreDNS pods.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSNode pool

   Contributing factor

   Change during peak hours

   Detection

   Monitoring tools detected an increased error rate and degraded service performance at 13:42 UTC, and the tech team started investigating at 13:44 UTC.

   Remediation

   • rescheduled the affected CoreDNS pod onto a new EC2 instance
   • scaled up the CoreDNS deployment to more replicas so DNS load was spread evenly across all pods

   Action items

   • Add scale-in protection so CoreDNS pods are not terminated unexpectedly during peak load
   • Review the CoreDNS configuration so that vertical scaling of the pods works
   • Hold regular capacity reviews so the CoreDNS deployment is provisioned for peak load

   View case
2. IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletion

   Neon2025

   Symptoms

   On 16 May 2025, in Neon's Kubernetes cluster in AWS us-east-1, new compute pods were scheduled but could not start and failed with IP allocation errors. Each Neon compute is a Postgres database running in a lightweight virtual machine in its own pod, so without new pods, idle customer databases could not be woken up.

   IP address exhaustionPods Pending

   Trigger

   On 16 May the Neon control plane's periodic job that terminates idle computes started failing. Compute creations continued, and the number of active computes rose from about 5k to about 8.1k within minutes. On 19 May, a follow-up change that set WARM_IP_TARGET back to 1 started the second round of failures.

   Root cause

   Capacity exhaustion

   Component

   AWS VPC CNIVPC subnet or pod address rangeCluster autoscaler

   Contributing factor

   Unreviewed defaultsNo capacity planningUnverified assumptionHarmful manual actionIneffective rollbackRetry storm

   Detection

   As cluster-autoscaler added nodes, the team saw IP allocation errors on new pods that were scheduled but could not start. The rate of successful pod starts and VPC subnet metrics were then used to follow the problem.

   Remediation

   • Set WARM_IP_TARGET=1 to release IPs held by nodes without free CPU or memory
   • Fixed the control plane's periodic job so idle computes were terminated again
   • Restarted the aws-node DaemonSet several times, each with a temporary improvement
   • Set WARM_IP_TARGET=0, MINIMUM_IP_TARGET=0 and WARM_ENI_TARGET=0, which let nodes allocate enough IPs and stabilised the cluster
   • On 19 May, set WARM_IP_TARGET back to 0 after setting it to 1 failed again
   • Enlarged the pool of pre-created warm compute pods to absorb failed pod starts

   Action items

   • Open a pull request to AWS CNI to improve how ipamd releases IPs in this situation
   • Follow up with AWS Support about the roughly 100 subnet IPs that were never allocated, for reasons Neon could not explain
   • Change the architecture to support horizontal scaling

   Lessons

   With default settings AWS CNI can keep many IPs on nodes that cannot use them, so the share of subnet IPs assigned to pods does not tell how close the subnet is to running out. WARM_IP_TARGET and WARM_ENI_TARGET follow different logic, a warm IP target of zero switches to the ENI logic, and IPs in cooldown count toward the target, which behaves badly under high pod churn. Changing these settings during an incident without knowing that logic prolonged the outage, and reverting a setting did not bring back the earlier healthy state.

   View case
3. IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes

   2i2c2025

   Symptoms

   On 12 May 2025 at 20:15 EEST a Freshdesk ticket reported that users' servers on the Earthscope hub were taking more than 15 minutes to start and that their kernels were being killed. It happened during a workshop and also affected other instructors using the hub; a second report said a short course of about 30 users was having major problems.

   Pod eviction

   Trigger

   A workshop with about 30 users on a profile option that guaranteed 8 GB of memory and set no memory limit on 32 GB nodes.

   Root cause

   Capacity exhaustion

   Component

   Container memory limitNode pool

   Contributing factor

   No resource requests or limits

   Detection

   A Freshdesk support ticket from the community at 20:15 EEST, brought to the team's attention in the #support-freshdesk Slack channel at 20:49.

   Remediation

   • ran a helm rollout and hub redeploy, which partly improved server starts
   • made the 'medium' profile available and raised its memory guarantee to reduce overcommitment
   • pre-warmed the cluster with 15 nodes for the 30-user workshop
   • set explicit memory limits, with the guarantee equal to the limit, on the community's small-node profile options (https://github.com/2i2c-org/infrastructure/pull/6047)

   Action items

   • Write an incident report
   • Hold an internal retrospective about the incident (https://github.com/2i2c-org/meta/issues/2194)

   Lessons

   A memory guarantee without a limit lets a user server grow into memory that other pods on the same node were counting on, and on a shared node that ends with users' pods evicting each other. Setting the guarantee equal to the limit removes the overcommitment. For a known workshop, pre-warming enough nodes ahead of time keeps server start times short.

   View case
4. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services

   Backpackr2025

   Symptoms

   During a Lunar New Year event in January 2024, a developer posted a message in Slack suspecting an outage. While several people were checking it, the problem grew into a full outage of Backpackr's web and app services, and the company decided to stop the event.

   Service outage

   Trigger

   An unexpected traffic surge, more than three times normal, during the Lunar New Year event in January 2024.

   Root cause

   Capacity exhaustion

   Component

   conntrack tableIngress controllerIstio

   Contributing factor

   Single point of failureUnreviewed defaultsMonitoring blind spot

   Detection

   A developer reported a suspected outage in Slack; the outage became total while several people were still confirming it.

   Remediation

   • stopped the Lunar New Year event

   Action items

   • Raise net.netfilter.nf_conntrack_max on all nodes to memory bytes/16384/2, and three times that on Istio nodes, applied by the nodeOptimizer DaemonSet
   • Configure Istio mesh networking so sidecars communicate directly instead of through the ingressGateway
   • Upgrade the Istio nodes from c6.xlarge (12.5 Gbps) to n-type instances with up to 25 Gbps of network bandwidth
   • Strengthen monitoring with ENA network bandwidth and packet drops, detailed Istio ingressGateway and sidecar metrics, and CPU, memory and saturation

   Lessons

   When every request in a cluster goes through one ingress gateway node, that node's connection tracking table and network bandwidth set the limit for the whole service. The default nf_conntrack table size was too small for three times normal traffic, and once it filled, packets were dropped. After the outage Backpackr strengthened its monitoring of network bandwidth, packet drops and the Istio components.

   View case
5. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down

   Backpackr2025

   Symptoms

   Services on the production Amazon EKS cluster behind the idus marketplace threw intermittent errors whose cause was hard to pin down. Calls to external services such as RDS and Redis failed now and then, and whenever DNS traffic spiked, domain lookups answered slowly or failed outright.

   DNS resolution failuresIntermittent connection failuresHigh latencyOOMKilledService outage

   Trigger

   All CoreDNS pods were scheduled onto the same node, and that node's memory usage spiked until it ran out of memory.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSClient-side DNS resolver and cachekubeletContainer memory limitNode pool

   Contributing factor

   Unreviewed defaultsSingle point of failureNo caching

   Detection

   The write-up does not say how the full outage was noticed. The earlier DNS trouble showed up as intermittent service errors, and investigating failed calls to RDS and Redis led to DNS resolution failures.

   Remediation

   • deployed NodeLocal DNSCache as a DaemonSet in front of CoreDNS
   • added a PodDisruptionBudget (maxUnavailable 1) and topologySpreadConstraints across zones for CoreDNS
   • lowered ndots from 5 to 2 with dnsConfig in the Deployments

   Lessons

   The CoreDNS setup that EKS creates by default (two pods) can be enough for a small or development cluster but becomes a bottleneck in a large production cluster, and DNS is a single point of failure that needs high availability and room to scale. Tuning the CoreDNS cache and buffer size only relieved the symptoms; the real load came from the extra queries that ndots:5 generates for external names. A CoreDNS pod without a memory limit can share a node's fate with everything else on it.

   View case

By root cause1