Cases2 cases

1 to 2 of 2

1. IL-0433 Shifting API traffic from EC2 to EKS on Fargate overloaded two CoreDNS pods and caused mass name-resolution errors

   MonotaRO2023

   Symptoms

   When MonotaRO raised the share of requests for the remaining endpoints of its backend API sent to EKS from 1% to 25%, Sentry filled with errors saying the API could not reach the database and outside services. The errors were about DNS name resolution, and the traffic had to be shifted back to the EC2 version.

   DNS resolution failuresHigh error rate

   Trigger

   MonotaRO raised the share of requests for the remaining API endpoints routed by the ALB to the EKS version from 1% to 25%.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSClient-side DNS resolver and cacheContainer CPU limitHorizontal pod autoscalerPod termination

   Contributing factor

   Unreviewed defaultsNo resource requests or limitsUnrepresentative stagingUnverified assumption

   Detection

   Sentry showed a large number of errors about connecting to the database and outside services right after the traffic share was raised.

   Remediation

   • Shifted the requests back to the EC2 version to stop the errors
   • Added an HPA to autoscale the CoreDNS pods as a stopgap, which let the migration to EKS finish

   Action items

   • Set ndots to 1 through dnsConfig in the application Deployments
   • Set cache 600 in the Corefile in the coredns ConfigMap to lengthen the CoreDNS cache TTL
   • Add lameduck 30s to the CoreDNS health plugin so CoreDNS keeps serving while it is scaled in
   • Consider reducing the differences between production and staging, weighed against cost

   Lessons

   MonotaRO assumed that a managed service would not cause problems and so skipped researching other companies' migrations, load testing and monitoring. Staging differed from production in its network layout, so ndots differed between them, and staging databases could not take a high-load test. Looking at the CoreDNS logs and resolv.conf instead of guessing found that ndots:5 made lookups of outside names try every search domain first.

   View case
2. IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percent

   Temporal2023

   Symptoms

   Request latency from the Temporal Server History service, the service that starts workflow executions and records their progress, was higher and less steady than it should be on a self-hosted Kubernetes deployment. The CPU dashboard showed the history pods using less than 60 percent of their CPU, yet the same pods were being throttled.

   CPU throttlingHigh latency

   Trigger

   Normal load on History pods that had a CPU limit of 2 cores on 8-core nodes, with the Go runtime's GOMAXPROCS left at its default of one per node core.

   Root cause

   Capacity exhaustion

   Component

   Container CPU limitLanguage runtime settingsPrometheus

   Contributing factor

   Unreviewed defaultsMonitoring blind spot

   Detection

   A Prometheus query for CPU throttling showed the history pods throttled although their CPU usage read under 60 percent.

   Remediation

   • set the GOMAXPROCS environment variable to 2 to match the 2-core CPU limit

   Action items

   • Set GOMAXPROCS automatically from the Kubernetes CPU limit, from Temporal release 1.21.0

   Lessons

   A CPU usage graph averaged over 15 seconds or more can sit far below 100 percent while a container is being throttled, so throttling needs its own metric. A Go program in a container should have GOMAXPROCS matched to its CPU limit, or to its CPU request when it has no limit. The author recommends not using CPU limits at all.

   View case

By root cause1