Incident Library
Applied filters ComponentCNI pluginSymptomDNS resolution failuresOrganisationTinderClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0472 Node-to-pod traffic and DNS failed on Tinder's 605-node Flannel cluster when every node's ARP cache passed gc_thresh3
    Tinder2019
    Symptoms

    In the early morning of 8 January 2019 Tinder's platform had a persistent outage. Communication from nodes to pods failed, and so did DNS lookups, because DNS ran inside the Kubernetes cluster.

    Service outageNetwork unreachableDNS resolution failures
    Trigger

    Earlier that morning, pod and node counts on the cluster were scaled up in response to an unrelated increase in platform latency.

    Root cause
    Capacity exhaustion
    Component
    Linux kernelCNI plugin
    Contributing factor
    Unreviewed defaults
    Remediation
    • Raised gc_thresh1, gc_thresh2 and gc_thresh3 on the nodes
    • Restarted Flannel so it re-registered the missing networks
    Lessons

    Tinder lists the outage among its learnings about network fabric limits: with Flannel over VXLAN, every node, and every pair of nodes that exchange pod traffic, adds entries to each node's ARP table, so at 605 nodes the default gc_thresh3 hard cap was exceeded on all nodes.

    View case
Check your cluster for this

This case involves CNI plugin and Linux kernel. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
CNI pluginLinux kernel
Contributing factor
Unreviewed defaults
Organisation
Tinder
  1. IL-0472 Node-to-pod traffic and DNS failed on Tinder's 605-node Flannel cluster when every node's ARP cache passed gc_thresh3
    Tinder2019