Cases2 cases

1 to 2 of 2

1. IL-0080 kube-dns on Mercari's GKE clusters could not keep up with peak-hour DNS query rates, and a Sev1 production outage followed when several services failed DNS resolution

   Mercari2025sev1

   Symptoms

   During peak hours, when Mercari's GKE clusters handle very high request rates, the platform team saw kube-dns reaching its limits. DNS query rates were high enough to overwhelm the kube-dns service.

   DNS resolution failuresIntermittent timeoutsService outage

   Trigger

   Peak-hour traffic on the GKE clusters produced more DNS queries than the few kube-dns pods could answer.

   Root cause

   Capacity exhaustion

   Component

   kube-dns

   Contributing factor

   No cachingSingle point of failure

   Detection

   During peak hours the team saw frequent NXDOMAIN and truncated-response errors and recurring "failed to refresh DNS cache" errors on the clusters, which showed that kube-dns had reached its limits.

   Remediation

   • deployed NodeLocal DNSCache on the Laboratory GKE cluster with a specific nodeAffinity to measure its effect
   • rolled NodeLocal DNSCache out gradually to the Dev and Prod environments by labelling node pools so its pods could be deployed there

   Lessons

   When every pod in a busy cluster resolves names through a few central kube-dns pods, DNS becomes a shared bottleneck, and a DNS failure quickly turns into higher latency, timeouts and cascading outages across microservices. A cache on each node takes most of that load off kube-dns and keeps resolution local to the node. Rolling the cache out first to an internal test cluster with nodeAffinity, and then node pool by node pool through labels, kept the blast radius of the change small.

   View case
2. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady

   Scaleway2024

   Symptoms

   At 23:48 CET on 24 January 2024 the Scaleway Kubernetes API for the FR-PAR region ran out of memory. At the same minute Scaleway saw requests to its API Gateway spike.

   Node NotReadyOOMKilledIntermittent timeoutsMonitoring outage

   Trigger

   Simultaneous deployments in the FR-PAR region pushed the Scaleway Kubernetes API out of memory.

   Root cause

   Capacity exhaustion

   Component

   Container memory limitkubelet

   Contributing factor

   Single point of failureRetry storm

   Detection

   Increased load on the API Gateway was detected at 23:48, the minute the incident began, and the incident was escalated to on-call engineers at 00:00:10.

   Remediation

   • raised the memory limit and garbage collection thresholds of the Kubernetes managed services
   • deployed additional replicas of key services
   • purged the registry cache and monitored the clusters as they stabilised
   • adjusted service configurations after the incident to keep the platform stable

   Action items

   • Implement a local authentication cache
   • Further develop node authentication
   • Enhance monitoring, adjust rate limiting strategies and improve incident response protocols

   Lessons

   Because the Kubernetes API also handled node authentication, its memory exhaustion made some nodes NotReady, and the endless retries of those nodes added load on the API Gateway. Scaleway plans a local authentication cache and further work on node authentication.

   View case

By root cause1