Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

No capacity planning

용량 계획 부재 · no-capacity-planning

7 cases · 2019 to 2025

Root cause Capacity exhaustion6 Cloud provider outage1

Component Kubernetes API server2 VPC subnet or pod address range2 AWS VPC CNI2 In-memory cache1 Cluster autoscaler1

Symptom Control plane lag2 Container crash loop2 High error rate2 High latency2

Organisation Adevinta1 BC Government1 Civis Analytics1 LINE+1 loveholidays

Cases

IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restartBC Government · 2025 · sev1
Root cause Capacity exhaustion
IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletionNeon · 2025
Root cause Capacity exhaustion
IL-0007 The AWS VPC CNI could not assign IP addresses to pods as an EKS cluster ran out of IPv4 space

View on the graph

conntrack table1

Persistent volumes and stateful sets1

Telemetry agent1

Intermittent connection failures2

IP address exhaustion2

Service outage2

Connection resets1

Control plane unresponsive1

DNS resolution failures1

Monitoring outage1

Probe failures1

Persistent volume full1

1

Adevinta · 2024

Root cause Capacity exhaustion

IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPUNamespace · 2024

Root cause Cloud provider outage

IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999LINE · 2020

Root cause Capacity exhaustion

IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020

Root cause Capacity exhaustion

IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the APICivis Analytics · 2019

Root cause Capacity exhaustion