Incident Library

Organisation

LINE+

LINE+ · line

2 cases · 2020 to 2022

Root cause Capacity exhaustion1Configuration change1
Component Cloud load balancer1conntrack table1HAProxy1Ingress controller1kubelet1Linux kernel1Prometheus1
Contributing factor Unverified assumption1Excessive blast radius1Change with a delayed effect1No capacity planning1No resource requests or limits1Retry storm1
Symptom Connection resets1Container crash loop1Database overload1High latency1Intermittent connection failures1Monitoring outage1OOMKilled1Probe failures1

Cases

  1. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on NginxLINE+ · 2022

    Root cause Configuration change

  2. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999LINE · 2020

    Root cause Capacity exhaustion