Incident Library

IL-0472 Tinder · 2019-01-08kubernetesawsnetworking

Node-to-pod traffic and DNS failed on Tinder's 605-node Flannel cluster when every node's ARP cache passed gc_thresh3

Tinder ran its platform on a Kubernetes cluster on Amazon EC2, provisioned with kube-aws, with Flannel as the network fabric forwarding packets over VXLAN. Each worker node takes its own /24 of pod addresses out of a larger /9 block, which adds one route table entry, one ARP entry on the flannel.1 interface and one forwarding database entry per node. Node-to-pod and pod-to-pod traffic also flows over the eth0 interface and adds an ARP entry for each source and destination node. This traffic was common: each Kubernetes Service had an ELB that registered every node, and a node that received a packet from the ELB used its iptables rules for the Service to pick a pod at random, often on another node.

On the morning of 8 January 2019 pod and node counts were scaled up in response to an unrelated latency increase. With 605 nodes in the cluster, the ARP entries exceeded the default gc_thresh3 on all nodes. Packets were dropped and entire Flannel /24 ranges disappeared from the ARP table, so node-to-pod communication and DNS lookups, served inside the cluster, failed. The fix was to raise gc_thresh1, gc_thresh2 and gc_thresh3 and restart Flannel so it registered the missing networks again.

Severity
–
Scope
Tinder's platform had a persistent outage: node-to-pod communication and in-cluster DNS lookups failed across the 605-node cluster
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

In the early morning of 8 January 2019 Tinder's platform had a persistent outage. Communication from nodes to pods failed, and so did DNS lookups, because DNS ran inside the Kubernetes cluster.

The ARP cache on every node of the 605-node cluster was exhausted. gc_thresh3 is the Linux kernel's hard cap on the ARP cache: "neighbor table overflow" log entries mean that even after a synchronous garbage collection there was no room for the neighbor entry, and the kernel then drops the packet. Besides the dropped packets, whole Flannel /24 pod address ranges went missing from the ARP table.

The outage followed a scale-up of pod and node counts on the cluster earlier that morning, made in response to an unrelated rise in platform latency.

Symptom Service outageNetwork unreachableDNS resolution failures

Root cause

Root cause Capacity exhaustion

Component Linux kernelCNI plugin
Contributing factor Unreviewed defaults
Trigger
Earlier that morning, pod and node counts on the cluster were scaled up in response to an unrelated increase in platform latency.

Detection

–

Remediation

Action items

–

Lessons

Tinder lists the outage among its learnings about network fabric limits: with Flannel over VXLAN, every node, and every pair of nodes that exchange pod traffic, adds entries to each node's ARP table, so at 605 nodes the default gc_thresh3 hard cap was exceeded on all nodes.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Linux kernel, CNI plugin among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.