Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

Horizontal pod autoscaler

수평 파드 오토스케일러 · hpa

7 cases · 2019 to 2026

Root cause Capacity exhaustion3 Cloud provider outage1 Configuration change1 DNS or network fault1 Software defect1

Contributing factor Monitoring blind spot3 Automation amplification2 Ignored early signal2 Unverified assumption1

Symptom Autoscaler misbehaviour4 High latency3 High error rate2 Service outage2

Organisation Airbnb1 Buildkite1 GitLab1 Kevin1 Lotte E-commerce1

Cases, by severity

IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026
Root cause Capacity exhaustion
IL-0181 A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed RunningKevin · 2026
Root cause Capacity exhaustion
Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation

View on the graph

Unreviewed defaults1

Change with a delayed effect1

Harmful manual action1

Overlapping failures1

Unrepresentative staging1

Scale-up failure1

CPU throttling1

Deploys failing1

DNS resolution failures1

Intermittent connection failures1

Intermittent timeouts1

Stale endpoint routing1

Throughput degradation1

IL-0182

이언철 · 2024

Root cause Capacity exhaustion

IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALBLotte E-commerce · 2022

Root cause DNS or network fault

IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutesGitLab · 2021

Root cause Configuration change

IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two podsAirbnb · 2020

Root cause Software defect

IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPAloveholidays · 2019

Root cause Cloud provider outage