Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Intermittent connection failures

간헐적 연결 실패 · intermittent-connection-failures

15 cases · 2018 to 2026

Root cause Software defect6 Capacity exhaustion4 DNS or network fault2 Cloud provider outage1 Configuration change1 Faulty deployment1

Component kubelet4 Application database3 HAProxy3 Ingress controller3 Linux kernel

Contributing factor Unverified assumption6 Automation amplification5 Unreviewed defaults3 Monitoring blind spot3

Organisation Amazon Web Services1 Backpackr1 DFDS1 DigitalOcean1 LINE+

Cases

IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hoursMicrosoft Azure · 2026
Root cause Faulty deployment
IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025
Root cause Software defect

View on the graph

3

Cloud load balancer2

conntrack table2

Client-side DNS resolver and cache2

Kubernetes API server2

Pod scheduling and placement2

In-memory cache1

Configuration templating1

Container CPU limit1

Authoritative DNS servers and records1

Horizontal pod autoscaler1

Node launch template1

Container memory limit1

Node operating system image1

Pod termination1

No capacity planning2

Overlapping failures2

Single point of failure2

Excessive blast radius1

Change during peak hours1

Change with a delayed effect1

Ignored early signal1

Unpatched known bug1

Harmful manual action1

No alert on limit1

No health probes1

No staged rollout1

Health check tied to a dependency1

Unrepresentative staging1

1

Microsoft Azure1

Toyota Connected1

IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025

Root cause Capacity exhaustion

IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPUNamespace · 2024

Root cause Cloud provider outage

IL-0154 Line cards on an SFO3 core router rebooted together and halved backbone capacity, and DOKS nodes whose guest networking did not recover stayed NotReady with degraded API servers until all affected nodes in the SFO3 DOKS fleet were recycledDigitalOcean · 2024

Root cause DNS or network fault

IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation이언철 · 2024

Root cause Capacity exhaustion

IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022

Root cause Software defect

IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on NginxLINE+ · 2022

Root cause Configuration change

IL-0056 Calls from EKS pods to services behind an NLB in another VPC failed intermittently because the AWS VPC CNI did source NAT on the nodeMindTickle · 2020

Root cause DNS or network fault

IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeoutsOmio · 2020

Root cause Software defect

IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020

Root cause Capacity exhaustion

IL-0062 Intermittent 502 errors behind a GKE Ingress because pods shut down before the load balancer stopped sending them trafficRavelin · 2019

Root cause Software defect

IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP orderSkyscanner · 2019

Root cause Software defect

IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019

Root cause Capacity exhaustion

IL-0050 Pods on Kubernetes failed to resolve an external Redis hostname because of ndots:5, the musl resolver in Alpine images, and Netty's DNS resolverToyota Connected · 2018

Root cause Software defect