Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Connection resets

연결 리셋 · connection-resets

2 cases · 2020 to 2022

Root cause Capacity exhaustion1 Configuration change1

Component conntrack table2 HAProxy2 Cloud load balancer1 Ingress controller1 kube-proxy1

Contributing factor Unverified assumption1 Excessive blast radius1 Change with a delayed effect1 Unknown limit1

Organisation LINE+1 loveholidays1

Cases

IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on NginxLINE+ · 2022
Root cause Configuration change
IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020
Root cause Capacity exhaustion

View on the graph

No alert on limit1

No capacity planning1