Incident Library

Organisation

Backpackr

Backpackr · backpackr

2 cases · 2025

Root cause Capacity exhaustion2
Component conntrack table1CoreDNS1Client-side DNS resolver and cache1Ingress controller1Istio1kubelet1Container memory limit1Node pool1
Contributing factor Unreviewed defaults2Single point of failure2Monitoring blind spot1No caching1
Symptom Service outage2DNS resolution failures1High latency1Intermittent connection failures1OOMKilled1

Cases

  1. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app servicesBackpackr · 2025

    Root cause Capacity exhaustion

  2. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025

    Root cause Capacity exhaustion