Cases3 cases

1 to 3 of 3

1. IL-0181 A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed Running

   Kevin2026

   Symptoms

   An e-commerce platform suddenly became slow to respond, worst on the product lookup page, and users began reporting that the page would not open.

   CPU throttlingHigh latency

   Trigger

   A post on the external community Ppomppu linked to the platform and brought a large number of users in a short time, who called the product lookup API heavily.

   Root cause

   Capacity exhaustion

   Component

   Container CPU limitHorizontal pod autoscaler

   Contributing factor

   Monitoring blind spot

   Detection

   Users reported that the product page would not open. The dashboard showed all pods Running, and the namespace CPU usage near 100% and the CPU usage metrics pointed to throttling.

   Remediation

   • raised the HPA minReplicas from 5 to 10
   • raised the container CPU limit from 1 core to 2 cores

   Action items

   • Keep adjusting the CPU limit to a suitable level through monitoring

   Lessons

   CPU throttling is hard to notice: pods stay Running, nothing restarts and the logs show no clear error, so a pod list does not reveal it and CPU usage metrics do. Settings that had handled expected event traffic became the bottleneck as soon as one external post multiplied the traffic. The author concludes that the team needs more preparation for traffic pattern changes and more thought about resource settings.

   View case
2. IL-0433 Shifting API traffic from EC2 to EKS on Fargate overloaded two CoreDNS pods and caused mass name-resolution errors

   MonotaRO2023

   Symptoms

   When MonotaRO raised the share of requests for the remaining endpoints of its backend API sent to EKS from 1% to 25%, Sentry filled with errors saying the API could not reach the database and outside services. The errors were about DNS name resolution, and the traffic had to be shifted back to the EC2 version.

   DNS resolution failuresHigh error rate

   Trigger

   MonotaRO raised the share of requests for the remaining API endpoints routed by the ALB to the EKS version from 1% to 25%.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSClient-side DNS resolver and cacheContainer CPU limitHorizontal pod autoscalerPod termination

   Contributing factor

   Unreviewed defaultsNo resource requests or limitsUnrepresentative stagingUnverified assumption

   Detection

   Sentry showed a large number of errors about connecting to the database and outside services right after the traffic share was raised.

   Remediation

   • Shifted the requests back to the EC2 version to stop the errors
   • Added an HPA to autoscale the CoreDNS pods as a stopgap, which let the migration to EKS finish

   Action items

   • Set ndots to 1 through dnsConfig in the application Deployments
   • Set cache 600 in the Corefile in the coredns ConfigMap to lengthen the CoreDNS cache TTL
   • Add lameduck 30s to the CoreDNS health plugin so CoreDNS keeps serving while it is scaled in
   • Consider reducing the differences between production and staging, weighed against cost

   Lessons

   MonotaRO assumed that a managed service would not cause problems and so skipped researching other companies' migrations, load testing and monitoring. Staging differed from production in its network layout, so ndots differed between them, and staging databases could not take a high-load test. Looking at the CoreDNS logs and resolv.conf instead of guessing found that ndots:5 made lookups of outside names try every search domain first.

   View case
3. IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percent

   Temporal2023

   Symptoms

   Request latency from the Temporal Server History service, the service that starts workflow executions and records their progress, was higher and less steady than it should be on a self-hosted Kubernetes deployment. The CPU dashboard showed the history pods using less than 60 percent of their CPU, yet the same pods were being throttled.

   CPU throttlingHigh latency

   Trigger

   Normal load on History pods that had a CPU limit of 2 cores on 8-core nodes, with the Go runtime's GOMAXPROCS left at its default of one per node core.

   Root cause

   Capacity exhaustion

   Component

   Container CPU limitLanguage runtime settingsPrometheus

   Contributing factor

   Unreviewed defaultsMonitoring blind spot

   Detection

   A Prometheus query for CPU throttling showed the history pods throttled although their CPU usage read under 60 percent.

   Remediation

   • set the GOMAXPROCS environment variable to 2 to match the 2-core CPU limit

   Action items

   • Set GOMAXPROCS automatically from the Kubernetes CPU limit, from Temporal release 1.21.0

   Lessons

   A CPU usage graph averaged over 15 seconds or more can sit far below 100 percent while a container is being throttled, so throttling needs its own metric. A Go program in a container should have GOMAXPROCS matched to its CPU limit, or to its CPU request when it has no limit. The author recommends not using CPU limits at all.

   View case

By root cause1