IL-0047 MindTickle · 2019-08-05 (published)kubernetesnetworking
MindTickle had moved most of its production workload to Kubernetes. After an earlier cluster-wide network failure that forced the team to build a new cluster and migrate every application, the team studied Kubernetes networking in depth, and in that period noticed intermittent delays in pod-to-pod calls. Because every delay was a multiple of five seconds, the team found that DNS lookups were the cause. The resolver in the C library (libc, or musl on Alpine) sends the A (IPv4) and AAAA (IPv6) queries on the same port; a race condition in conntrack, which shows up where DNAT or SNAT translation happens, sometimes loses one of them, and the resolver treats the whole lookup as timed out and retries after 5 seconds.
Adding the resolver option single-request-reopen to the pods' dnsConfig made the resolver use a different socket for the second query, and tcpdump confirmed the A and AAAA queries now left from different ports. The delays disappeared for ordinary Linux images. gRPC services still hit DEADLINE_EXCEEDED, and tcpdump showed their A and AAAA queries still sharing a port: gRPC uses its own c-ares resolver when it is available, so the libc option had no effect. Setting GRPC_DNS_RESOLVER=native in the Dockerfiles made gRPC use the system resolver, and the problem was gone there too. Alpine images were left unchanged because most of the applications use other Linux images.
Pod-to-pod calls were intermittently slow and the web application felt slow overall. A request that pod A sent at time x reached pod B at x + 5s, x + 10s or x + 15s, always a multiple of five seconds.
gRPC services showed it as DEADLINE_EXCEEDED errors when their timeout was 3 seconds.
A capture with tcpdump port 53 made the pattern visible: the pod sent the A and AAAA queries for a name from the same source port, the AAAA answers came back, and the A query for the same name was only sent again about five seconds later (18:09:09 to 18:09:15 in the published capture) before it was answered.
Symptom Intermittent timeoutsHigh latencyDNS resolution failures
Root cause DNS or network fault
The team noticed intermittent delays in pod-to-pod calls and slowness in the web application, and that the delays were always multiples of five seconds.
–
Delays that are exact multiples of five seconds point to DNS lookups being retried after the resolver's 5-second timeout. A resolver option set in the pod spec only helps processes that use the libc resolver; runtimes with their own resolver, such as gRPC with c-ares, ignore it.
IL-0057 Higher p99 latency after moving a reverse proxy to EKS, from outgoing connections delayed by a SNAT conntrack racePrezi · 2019same root cause · shared Linux kernel, conntrack table
IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNSAdevinta · 2023same root cause · shared Client-side DNS resolver and cache
IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALBLotte E-commerce · 2022same root cause · shared conntrack table
IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entriesPreply · 2020same root cause · shared conntrack table
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.