Incident Library
Applied filters ComponentHorizontal pod autoscalerSymptomHigh error rateFactorNo resource requests or limitsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0433 Shifting API traffic from EC2 to EKS on Fargate overloaded two CoreDNS pods and caused mass name-resolution errors
    MonotaRO2023
    Symptoms

    When MonotaRO raised the share of requests for the remaining endpoints of its backend API sent to EKS from 1% to 25%, Sentry filled with errors saying the API could not reach the database and outside services. The errors were about DNS name resolution, and the traffic had to be shifted back to the EC2 version.

    DNS resolution failuresHigh error rate
    Trigger

    MonotaRO raised the share of requests for the remaining API endpoints routed by the ALB to the EKS version from 1% to 25%.

    Root cause
    Capacity exhaustion
    Component
    CoreDNSClient-side DNS resolver and cacheContainer CPU limitHorizontal pod autoscalerPod termination
    Contributing factor
    Unreviewed defaultsNo resource requests or limitsUnrepresentative stagingUnverified assumption
    Detection

    Sentry showed a large number of errors about connecting to the database and outside services right after the traffic share was raised.

    Remediation
    • Shifted the requests back to the EC2 version to stop the errors
    • Added an HPA to autoscale the CoreDNS pods as a stopgap, which let the migration to EKS finish
    Action items
    • Set ndots to 1 through dnsConfig in the application Deployments
    • Set cache 600 in the Corefile in the coredns ConfigMap to lengthen the CoreDNS cache TTL
    • Add lameduck 30s to the CoreDNS health plugin so CoreDNS keeps serving while it is scaled in
    • Consider reducing the differences between production and staging, weighed against cost
    Lessons

    MonotaRO assumed that a managed service would not cause problems and so skipped researching other companies' migrations, load testing and monitoring. Staging differed from production in its network layout, so ndots differed between them, and staging databases could not take a high-load test. Looking at the CoreDNS logs and resolv.conf instead of guessing found that ndots:5 made lookups of outside names try every search domain first.

    View case
Check your cluster for this

This case involves CoreDNS and Container CPU limit. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
CoreDNSContainer CPU limitClient-side DNS resolver and cacheHorizontal pod autoscalerPod termination
Contributing factor
Unverified assumptionUnreviewed defaultsNo resource requests or limitsUnrepresentative staging
Organisation
MonotaRO
  1. IL-0433 Shifting API traffic from EC2 to EKS on Fargate overloaded two CoreDNS pods and caused mass name-resolution errors
    MonotaRO2023