Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

Retry storm

재시도 폭주 · retry-storm

16 cases · 2019 to 2026

Root cause Capacity exhaustion10 Configuration change3 Software defect2 Faulty deployment1

Component Application database5 In-memory cache4 Container memory limit4 Cloud load balancer2

Symptom High latency8 High error rate7 OOMKilled6 Service outage6

Organisation Scaleway2 AdGuard VPN1 Amazon Web Services1 Buildkite1 Fullstaq

Cases

IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hoursMicrosoft Azure · 2026
Root cause Faulty deployment
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026
Root cause Capacity exhaustion

View on the graph

Cluster autoscaler2

Kubernetes API server2

Persistent volumes and stateful sets2

Cloud controller manager1

conntrack table1

Client-side DNS resolver and cache1

Authoritative DNS servers and records1

GKE control plane1

Horizontal pod autoscaler1

Ingress controller1

Cluster authentication provider1

Pod priority and preemption1

Pod scheduling and placement1

VPC subnet or pod address range1

Database overload4

DNS resolution failures3

Intermittent connection failures3

Monitoring outage3

Control plane lag2

Container crash loop2

Intermittent timeouts2

Probe failures2

Autoscaler misbehaviour1

Scale-up failure1

Cloud API rate limit exhausted1

Connection resets1

Control plane unresponsive1

IP address exhaustion1

Sustained high node CPU1

Outbound connection exhaustion1

1

Microsoft Azure1

IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks

AdGuard VPN · 2025

Root cause Configuration change

IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025

Root cause Capacity exhaustion

IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025

Root cause Software defect

IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletionNeon · 2025

Root cause Capacity exhaustion

IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hoursMagic Pages · 2025

Root cause Capacity exhaustion

IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three timesHugging Face · 2024

Root cause Software defect

IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PARScaleway · 2024

Root cause Capacity exhaustion

IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024

Root cause Capacity exhaustion

IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1

Root cause Capacity exhaustion

IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on NginxLINE+ · 2022

Root cause Configuration change

IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worseGoogle Cloud · 2022

Root cause Capacity exhaustion

IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus downGrafana Labs · 2019

Root cause Configuration change

IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loopFullstaq · 2019

Root cause Capacity exhaustion

IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019

Root cause Capacity exhaustion