Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

Client-side DNS resolver and cache

클라이언트 쪽 DNS 리졸버와 캐시 · dns-resolver

10 cases · 2018 to 2025

Root cause Capacity exhaustion4 DNS or network fault2 Breaking change in an upgrade2 Configuration change1 Software defect1

Contributing factor Unreviewed defaults7 Unknown limit3 Unverified assumption2 Excessive blast radius2

Symptom DNS resolution failures10 High latency4 High error rate3 Intermittent timeouts3

Organisation Adevinta1 Airbnb1 Backpackr1 Microsoft Azure1 MindTickle

Cases, by severity

IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hoursWealthy · 2025 · sev1
Root cause Breaking change in an upgrade
IL-0048 DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookupsunknown · 2025
Root cause Capacity exhaustion
All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down

View on the graph

Monitoring blind spot2

No staged rollout2

Single point of failure2

Automation amplification1

Ignored early signal1

Cache-masked failure1

Limited debug access1

Ineffective rollback1

Deferred upgrades1

Unrepresentative staging1

Service outage3

Intermittent connection failures2

Monitoring outage2

Scale-up failure1

Control plane unresponsive1

Container crash loop1

Image pull failures1

Load imbalance1

1

Toyota Connected1

IL-0027

Backpackr · 2025

Root cause Capacity exhaustion

IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI servicesOpenAI · 2024

Root cause Configuration change

IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNSAdevinta · 2023

Root cause DNS or network fault

IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOffMicrosoft Azure · 2022

Root cause Breaking change in an upgrade

IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 secondsAirbnb · 2019

Root cause Capacity exhaustion

IL-0047 Pod-to-pod requests delayed by 5, 10 or 15 seconds because of DNS lookups lost to a conntrack raceMindTickle · 2019

Root cause DNS or network fault

IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019

Root cause Capacity exhaustion

IL-0050 Pods on Kubernetes failed to resolve an external Redis hostname because of ndots:5, the musl resolver in Alpine images, and Netty's DNS resolverToyota Connected · 2018

Root cause Software defect