Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Load imbalance

부하 불균형 · uneven-load

4 cases · 2019 to 2022

Root cause Capacity exhaustion1 Configuration change1 Dependency failure1 Software defect1

Component Configuration templating2 Service discovery2 Container runtime1 Credential broker for pods1

Contributing factor Unreviewed defaults2 Ignored early signal2 Unrepresentative staging2 Unverified assumption1

Organisation Airbnb1 Grafana Labs1 Skyscanner1 Target1

Cases

IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rolloutGrafana Labs · 2022
Root cause Configuration change
IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 secondsAirbnb · 2019
Root cause Capacity exhaustion
IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP order

View on the graph

Client-side DNS resolver and cache1

Ingress controller1

Container memory limit1

Telemetry agent1

Automation amplification1

Excessive blast radius1

Unpatched known bug1

Single point of failure1

Skyscanner · 2019

Root cause Software defect

IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip meshTarget · 2019 · sev1

Root cause Dependency failure