Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

HAProxy

HAProxy · haproxy

4 cases · 2019 to 2022

Root cause Configuration change2 Capacity exhaustion1 Software defect1

Contributing factor Unverified assumption2 Automation amplification1 Excessive blast radius1 Change with a delayed effect1

Symptom High latency3 Intermittent connection failures3 Connection resets2 High error rate2

Organisation GitLab1 LINE+1 loveholidays1 Skyscanner1

Cases, by severity

IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on NginxLINE+ · 2022
Root cause Configuration change
IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errorsGitLab · 2022
Root cause Configuration change

View on the graph

Ignored early signal1

No alert on limit1

No capacity planning1

Unrepresentative staging1

Database overload1

Intermittent timeouts1

Load imbalance1

IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resets

loveholidays · 2020

Root cause Capacity exhaustion

IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP orderSkyscanner · 2019

Root cause Software defect