Incident Library

Organisation

MonotaRO

MonotaRO · monotaro

1 case · 2023

Root cause Capacity exhaustion
Component CoreDNSContainer CPU limitClient-side DNS resolver and cacheHorizontal pod autoscalerPod termination
Contributing factor Unverified assumptionUnreviewed defaultsNo resource requests or limitsUnrepresentative staging
Symptom DNS resolution failuresHigh error rate

Cases

  1. IL-0433 Shifting API traffic from EC2 to EKS on Fargate overloaded two CoreDNS pods and caused mass name-resolution errorsMonotaRO · 2023

    Root cause Capacity exhaustion