Incident Library

IL-0461 Datadog · 2022-04-13 (published)kubernetesawsnetworking

Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS

Datadog's metrics query service runs on EC2 instances with two ENIs: ens5 for the host and ens6 for the pods. Cilium in ipam:eni mode gives each pod a routable VPC address on ens6, a route to its veth and an IP rule that sends its traffic out through ens6. When a rollout deletes a pod, Cilium removes the route and the rule, but the VPC keeps delivering traffic for that IP to ens6. A SYN sent to the deleted pod's IP then has no return path through ens6, so reverse path filtering treats it as a Martian packet and drops it before it reaches the Linux conntrack, and the client gets no SYN-ACK, RST or ICMP error. The hosts used loose mode, which should allow this asymmetric path; reading the kernel code after the incident, the team found that the check falls through to the drop because ens6 has no IP address assigned, and adding an unrelated address to ens6 made loose mode behave as expected.

Clients needed about 60 seconds to learn the new pod addresses: the service waits 10 seconds after GracefulStop, the external-dns version in use only updates the Route 53 record after the kubelet has deleted the pod and syncs every 15 seconds with a 15-second TTL, and the gRPC clients use the default min_time_between_resolutions of 30 seconds. In June 2021 Datadog had switched one service's clients from pick_first to round_robin to fix another issue and then made round_robin the default; the metrics query clients picked it up in an August 2021 rollout. Because the application creates one gRPC channel per backend, the policy did not change the number of connections, but with round_robin a channel reconnects automatically whenever it is disconnected, and an aggressive reconnect timeout set long before made each of about 900 clients retry about every 300 milliseconds, up to 30,000 SYN packets per second. Each unanswered SYN took an entry in the VPC conntrack, entries are only removed after 60 seconds without an answer, and the table filled. The drops broke node-local-dns connections to its upstream resolvers, which surfaced as DNS errors.

Datadog removed the custom gRPC reconnect parameters to restore the defaults, the next rollout showed none of the errors seen since September, and the incident was closed. Datadog later contributed a change to Cilium that adds an unreachable route for a deleted pod's IP, so incoming SYNs are not dropped as Martian packets and clients get a clear ICMP error. The investigation took months and involved an AWS support case.

Severity
–
Scope
Errors and added latency for dashboards and monitors that read from the metrics query service during its rollouts; client retries kept most errors from users
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

From September 2021, each rollout of an update to Datadog's metrics query service, which serves real-time and historical metric data to the web frontend and to monitor evaluation, came with a rise in errors. Clients retried failed queries, so few errors reached users, but the retries slowed dashboards and monitors. The service's logs showed DNS errors stopping it from connecting to the data stores it depends on.

In the cluster, the node-local-dns pods of NodeLocal DNSCache, a slimmed-down CoreDNS running as a DaemonSet, were being OOM killed with a 64 MB memory limit and were rejecting requests above their max_concurrent limit of 1,000 during rollouts. Raising the memory to 256 MB stopped the OOM kills, but the DNS errors continued. The request rate, about 400 queries per second and close to 2,000 during rollouts, was far below what node-local-dns should handle, and its metrics showed it marking the upstream DNS resolvers as unhealthy, which pointed to failing connections to the upstreams.

Network throughput stayed well below the 5-Gbps limit of the instance types, but TCP retransmits rose with each rollout. After ENA metrics were enabled in the Datadog Agent, the metric conntrack_allowance_exceeded rose during rollouts, showing packets dropped because the VPC connection tracking table was saturated, while the Linux conntrack on the same hosts peaked below 60,000 connections. Tests on larger instance types showed that scaling up the service removed the errors.

VPC Flow Logs showed that after a pod was deleted, clients that evaluate metric data kept sending SYN packets to its old IP address, and the SYNs got no answer: one old IP received about 90,000 connection attempts in about 90 seconds. On the old pod's node the kernel dropped these packets with "IPv4: martian source 10.x.y.z from 10.a.b.c, on dev ens6", and a route lookup returned "RTNETLINK answers: Invalid cross-device link".

Symptom High error rateDNS resolution failuresHigh latencyOOMKilledPacket lossStale endpoint routing

Root cause

Root cause Configuration change

Component conntrack tableCiliumLinux kernelCoreDNSClient-side DNS resolver and cacheexternal-dnsHTTP and RPC client library
Contributing factor Change with a delayed effectRetry stormUnknown limitMonitoring blind spot
Trigger
Rollouts of the metrics query service, which delete the old pods. The client behaviour behind the failures came from an August 2021 rollout of the metrics query clients that applied round_robin as the new default gRPC load balancing policy.

Detection

Errors rose during rollouts of the metrics query service, and the service's logs showed DNS errors when connecting to its data stores.

Remediation

Action items

Lessons

The errors looked like DNS but were not. ENA metrics and VPC Flow Logs gave the visibility into VPC connection tracking that host metrics did not, and Cilium, Kubernetes and AWS networking interact in ways that can make reverse path filtering drop packets it should allow. DNS propagation time depends on many settings. The causes were changes Datadog had made itself, to work around a dependency bug, to resolve earlier incidents and to override default reconnect parameters, which looked safe at the time and showed their effects much later, when the link was hard to see.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, conntrack table, Cilium, Linux kernel among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.