Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services
    Backpackr2025
    Symptoms

    During a Lunar New Year event in January 2024, a developer posted a message in Slack suspecting an outage. While several people were checking it, the problem grew into a full outage of Backpackr's web and app services, and the company decided to stop the event.

    Service outage
    Trigger

    An unexpected traffic surge, more than three times normal, during the Lunar New Year event in January 2024.

    Root cause
    Capacity exhaustion
    Component
    conntrack tableIngress controllerIstio
    Contributing factor
    Single point of failureUnreviewed defaultsMonitoring blind spot
    Detection

    A developer reported a suspected outage in Slack; the outage became total while several people were still confirming it.

    Remediation
    • stopped the Lunar New Year event
    Action items
    • Raise net.netfilter.nf_conntrack_max on all nodes to memory bytes/16384/2, and three times that on Istio nodes, applied by the nodeOptimizer DaemonSet
    • Configure Istio mesh networking so sidecars communicate directly instead of through the ingressGateway
    • Upgrade the Istio nodes from c6.xlarge (12.5 Gbps) to n-type instances with up to 25 Gbps of network bandwidth
    • Strengthen monitoring with ENA network bandwidth and packet drops, detailed Istio ingressGateway and sidecar metrics, and CPU, memory and saturation
    Lessons

    When every request in a cluster goes through one ingress gateway node, that node's connection tracking table and network bandwidth set the limit for the whole service. The default nf_conntrack table size was too small for three times normal traffic, and once it filled, packets were dropped. After the outage Backpackr strengthened its monitoring of network bandwidth, packet drops and the Istio components.

    View case
Check your cluster for this

This case involves conntrack table and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
conntrack tableIngress controllerIstio
Contributing factor
Unreviewed defaultsMonitoring blind spotSingle point of failure
Organisation
Backpackr
  1. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services
    Backpackr2025