Incident Library
Applied filters SymptomNetwork unreachableDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases8 cases

8 to 8 of 8

  1. IL-0472 Node-to-pod traffic and DNS failed on Tinder's 605-node Flannel cluster when every node's ARP cache passed gc_thresh3
    Tinder2019
    Symptoms

    In the early morning of 8 January 2019 Tinder's platform had a persistent outage. Communication from nodes to pods failed, and so did DNS lookups, because DNS ran inside the Kubernetes cluster.

    Service outageNetwork unreachableDNS resolution failures
    Trigger

    Earlier that morning, pod and node counts on the cluster were scaled up in response to an unrelated increase in platform latency.

    Root cause
    Capacity exhaustion
    Component
    Linux kernelCNI plugin
    Contributing factor
    Unreviewed defaults
    Remediation
    • Raised gc_thresh1, gc_thresh2 and gc_thresh3 on the nodes
    • Restarted Flannel so it re-registered the missing networks
    Lessons

    Tinder lists the outage among its learnings about network fabric limits: with Flannel over VXLAN, every node, and every pair of nodes that exchange pod traffic, adds entries to each node's ARP table, so at 605 nodes the default gc_thresh3 hard cap was exceeded on all nodes.

    View case
Check your cluster for this

These 8 cases involve Network control plane (SDN) and BGP routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause6

Cloud provider outage1 case

Component
Cloud controller managerCloud load balancerGKE control plane
Contributing factor
–
Organisation
Harness
  1. IL-0463 Secure Connect connectors timed out after a GKE control plane upgrade left Harness's internal load balancer unreachable
    Harness2026

Capacity exhaustion2 cases

Component
BGP routing1CNI plugin1Database1InfiniBand fabric1Linux kernel1Model hub1Network control plane (SDN)1NIC, cable and switch port link1
Contributing factor
Automation amplification1Excessive blast radius1Unreviewed defaults1
Organisation
IONOS Cloud1Tinder1
  1. IL-0350 Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast table
    IONOS Cloud2026
  2. IL-0472 Node-to-pod traffic and DNS failed on Tinder's 605-node Flannel cluster when every node's ARP cache passed gc_thresh3
    Tinder2019

Software defect2 cases

Component
BGP routing2Network control plane (SDN)2Database1Managed Kubernetes control plane and provider API1Network switch1
Contributing factor
Automation amplification1Excessive blast radius1Monitoring blind spot1Ineffective rollback1Gap in automated tests1
Organisation
Nebius1Scaleway1
  1. IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivity
    Nebius2025
  2. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck
    Scaleway2024

Operator error1 case

Component
CI systemTerraformVPC route tables and firewall rules
Contributing factor
No safeguard on destructive change
Organisation
UK Ministry of Justice
  1. IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline
    UK Ministry of Justice2024

Faulty deployment1 case

Component
Cloud load balancerNetwork control plane (SDN)
Contributing factor
Excessive blast radiusNo alert on limit
Organisation
Google Cloud
  1. IL-0530 Virtual router controllers in us-central1 ran out of memory in a loop, cutting networking for new and migrated VMs
    Google Cloud2023

Configuration change1 case

Component
Cloud load balancerCNI pluginNode pool
Contributing factor
Unverified assumptionNo safeguard on destructive changeNo staged rollout
Organisation
Jenkins project
  1. IL-0473 Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate them
    Jenkins project2023