Cases11 cases

4 to 8 of 11

1. IL-0433 Shifting API traffic from EC2 to EKS on Fargate overloaded two CoreDNS pods and caused mass name-resolution errors

   MonotaRO2023

   Symptoms

   When MonotaRO raised the share of requests for the remaining endpoints of its backend API sent to EKS from 1% to 25%, Sentry filled with errors saying the API could not reach the database and outside services. The errors were about DNS name resolution, and the traffic had to be shifted back to the EC2 version.

   DNS resolution failuresHigh error rate

   Trigger

   MonotaRO raised the share of requests for the remaining API endpoints routed by the ALB to the EKS version from 1% to 25%.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSClient-side DNS resolver and cacheContainer CPU limitHorizontal pod autoscalerPod termination

   Contributing factor

   Unreviewed defaultsNo resource requests or limitsUnrepresentative stagingUnverified assumption

   Detection

   Sentry showed a large number of errors about connecting to the database and outside services right after the traffic share was raised.

   Remediation

   • Shifted the requests back to the EC2 version to stop the errors
   • Added an HPA to autoscale the CoreDNS pods as a stopgap, which let the migration to EKS finish

   Action items

   • Set ndots to 1 through dnsConfig in the application Deployments
   • Set cache 600 in the Corefile in the coredns ConfigMap to lengthen the CoreDNS cache TTL
   • Add lameduck 30s to the CoreDNS health plugin so CoreDNS keeps serving while it is scaled in
   • Consider reducing the differences between production and staging, weighed against cost

   Lessons

   MonotaRO assumed that a managed service would not cause problems and so skipped researching other companies' migrations, load testing and monitoring. Staging differed from production in its network layout, so ndots differed between them, and staging databases could not take a high-load test. Looking at the CoreDNS logs and resolv.conf instead of guessing found that ndots:5 made lookups of outside names try every search domain first.

   View case
2. IL-0175 Lowering a JVM pod's CPU request from 2 to 1 pushed memory past the 90% alert because the JVM treats a cgroup v1 cpu.shares value of exactly 1024 as unset and sized Reactor Netty threads from the node's 36 CPUs

   Hyperconnect2022

   Symptoms

   During a DevOps review of resource usage across microservices, the team lowered resources.requests.cpu from 2 to 1 for notification-preferences, a Spring WebFlux service not yet used in production. A few minutes later an alarm fired because memory usage was above 90%. Neither the memory allocation nor the JVM's GC arguments had been changed.

   Memory pressure

   Trigger

   The team lowered resources.requests.cpu for the notification-preferences deployment from 2 to 1 as part of resource optimisation, with no CPU limit set.

   Root cause

   Software defect

   Component

   Language runtime settingsContainer CPU limit

   Contributing factor

   No resource requests or limits

   Detection

   A memory usage alarm above 90% fired a few minutes after the CPU request was lowered.

   Remediation

   • set -XX:ActiveProcessorCount=1 in JAVA_OPTIONS so the JVM uses one processor regardless of the cgroup values

   Action items

   • Write an internal notice about the issue

   Lessons

   On cgroup v1, a JVM in a pod with a CPU request of exactly one core and no CPU limit sees every CPU of the node, because cpu.shares 1024 is read as unset. Thread pools sized from availableProcessors(), such as Reactor Netty's I/O workers, grow accordingly and use more memory. When lowering requests for JVM services without a CPU limit, set -XX:ActiveProcessorCount or a CPU limit explicitly. Later JDKs ignore CPU requests for this calculation altogether.

   View case
3. IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub

   2i2c2020

   Symptoms

   At 08:52 PM IST a report came in from WER that many students could not access the JupyterHub and that it was frozen; pages were stuck and nothing was usable. A rise in activity was visible by 09:02 PM, but the usual fixes, such as opening an incognito window and restarting servers, did not help.

   Service outageSustained high node CPU

   Trigger

   WER students' pods used a good chunk of their memory limit (tests had shown they almost always use just under 1G of their 2G limit, against a 512M guarantee) on nodes shared with the hub's core pods.

   Root cause

   Capacity exhaustion

   Component

   Node poolContainer memory limit

   Contributing factor

   No resource requests or limitsUnverified assumptionExcessive blast radius

   Detection

   WER reported stuck pages and students unable to reach the hub in a GitHub issue (2i2c-org/docs#27).

   Remediation

   • removed memory overcommit for the WER students
   • raised resource guarantees for all core pods, which restarted them and moved some to a new node

   Action items

   • Put user pods in a separate pool so they do not create pressure on the core pods (infrastructure#89)
   • Set limits on the support infrastructure (Prometheus, Grafana, ingress) as well (infrastructure#90)
   • Document and think about overcommit ratios for memory usage (infrastructure#91)
   • Set up better Grafana dashboards to monitor resource usage (infrastructure#92)
   • Document how people can get kubectl access to the cluster so others can look into issues too (infrastructure#87)

   Lessons

   Core pods without resource guarantees get whatever is left on a node, so when user pods on the same nodes fill the CPU and memory, the hub that serves everyone freezes. The memory overcommit ratio has to come from measured use: WER students used just under 1G against a 512M guarantee, which a 4x overcommit could not hold.

   View case
4. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999

   LINE2020

   Symptoms

   One Monday morning the author opened the Prometheus dashboard to check a Kubernetes cluster for alerts and could not reach it. The Prometheus pod was in CrashLoopBackOff: kubectl get pods showed it 1/2 ready with 950 restarts over 3d20h on worker3.

   Container crash loopOOMKilledMonitoring outageProbe failures

   Trigger

   After start, the Prometheus time series count rose to 2,162,256 within five minutes and its memory use filled the 8GB host.

   Root cause

   Capacity exhaustion

   Component

   PrometheuskubeletLinux kernel

   Contributing factor

   No resource requests or limitsNo capacity planning

   Detection

   The author found it on a Monday morning when the Prometheus dashboard, opened to check the cluster for alerts, could not be reached.

   Remediation

   • removed most of the unnecessary settings from the Prometheus ConfigMap, after which the container stopped crashing

   Action items

   • Set appropriate resource requests and limits on the Prometheus pod
   • Reduce unneeded metrics by adjusting scrape targets or scrape interval
   • Estimate Prometheus memory needs, provide dedicated hosts with a node label such as deploy-topology: Prometheus, and pin the Prometheus pod to them with a nodeSelector

   Lessons

   Exit code 137 together with an OOM event in the kubelet log means the kernel killed the container, not kubelet. Under node memory pressure the kernel kills the process with the highest score, and kubelet derives each container's oom_score_adj from the pod's QoS class: a container without a memory request in a pod where another container has one ends up Burstable with a score near 1000, so it is killed first. In a large cluster, Prometheus memory needs to be estimated from the expected series count and requested explicitly.

   View case
5. IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limits

   Blue Matador2019

   Symptoms

   On a Saturday at 5:12 pm, Blue Matador's own monitoring raised a SystemOOM event for one node in the production Kubernetes cluster. Four minutes later a second warning reported a low EBS Burst Balance on that node's root volume. CloudWatch data showed the burst balance had actually bottomed out at 5:02 pm, before the memory event; the warning arrived late because EBS metrics lag 10 to 15 minutes.

   OOMKilled

   Trigger

   Memory use of the fluentd-sumologic log shipper pods grew until the node ran out of memory; log volume in Sumo Logic had been steady, so a burst of logs was not the cause.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agentContainer memory limit

   Contributing factor

   No resource requests or limitsIgnored early signal

   Detection

   Blue Matador's own monitoring product alerted on the SystemOOM event, followed minutes later by a low EBS Burst Balance warning for the same node.

   Remediation

   • restarted all fluentd-sumologic pods so they started with low memory
   • set RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR to 0.9 on the fluentd-sumologic pods
   • added memory requests of 128Mi and limits of 1024Mi, and CPU requests of 100m and limits of 250m, to the fluentd-sumologic pods

   Action items

   • Apply default resource limits and resource quotas so that every pod runs with requests and limits

   Lessons

   When checking for restarted pods, look in every namespace, not only the application one. Set resource requests and limits on every pod, third-party ones in particular, so that one workload cannot exhaust a node. Investigate side signals such as the disk I/O spike while the logs still exist.

   View case

By root cause4