Incident Library

Organisation

LINE Plus

LINE Plus · line-plus

1 case · 2022

Root cause Configuration change
Component Cloud load balancerconntrack tableHAProxyIngress controller
Contributing factor Unverified assumptionExcessive blast radiusChange with a delayed effectRetry storm
Symptom Connection resetsDatabase overloadHigh latencyIntermittent connection failures

Cases

  1. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on NginxLINE Plus · 2022

    Root cause Configuration change