Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Intermittent timeouts

간헐적 타임아웃 · intermittent-timeouts

21 cases · 2019 to 2026

Root cause Software defect7 Capacity exhaustion5 Configuration change3 DNS or network fault3 Faulty deployment1 Operator error1 Breaking change in an upgrade1

Component Linux kernel5 kubelet4 Cloud load balancer3 conntrack table3

Contributing factor Unreviewed defaults7 Monitoring blind spot4 Automation amplification3 Unknown limit3

Organisation Microsoft Azure2 Airbnb1 Cresta1 DFDS1 DigitalOcean1

Cases

IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second burstsMUSINSA · 2026
Root cause Configuration change
IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hoursMicrosoft Azure · 2026
Root cause Faulty deployment

View on the graph

Client-side DNS resolver and cache3

Ingress controller3

Node operating system image3

Application database2

Container memory limit2

In-memory cache1

Container runtime1

Container CPU limit1

Horizontal pod autoscaler1

Model inference server1

Node launch template1

Application logging library1

Pod termination1

Telemetry agent1

VPC DNS resolver1

Excessive blast radius2

Unpatched known bug2

No safeguard on destructive change2

Overlapping failures2

Single point of failure2

Unverified assumption1

Change during peak hours1

No alert on limit1

No health probes1

No resource requests or limits1

No staged rollout1

Partial rollout1

Health check tied to a dependency1

Deferred upgrades1

Lotte E-commerce1

IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodesDigitalOcean · 2026

Root cause Capacity exhaustion

IL-0048 DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookupsunknown · 2025

Root cause Capacity exhaustion

IL-0080 kube-dns on Mercari's GKE clusters could not keep up with peak-hour DNS query rates, and a Sev1 production outage followed when several services failed DNS resolutionMercari · 2025 · sev1

Root cause Capacity exhaustion

IL-0201 A framework upgrade caused kernel panics in GitHub's Kubernetes infrastructure, and GitHub.com web and API requests, mostly for issues and pull requests, failed intermittently for about 16 hoursGitHub · 2025

Root cause Breaking change in an upgrade

IL-0081 API requests timed out during deployment rollouts because the ingress-nginx enable-serial-reloads option held back endpoint updates while a long NGINX reload was runningCresta · 2025

Root cause Software defect

IL-0131 A new kind of GenerateContent request made Vertex AI prediction API servers segfault, and once enough of them restarted at the same time the serving fleet could not carry the loadGoogle Cloud · 2024

Root cause Software defect

IL-0161 A faulty earlier API update made new LKE cluster provisioning in Toronto time out for three days, leaving clusters with no node pools or the wrong number of nodesLinode · 2024

Root cause Software defect

IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024

Root cause Capacity exhaustion

IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022

Root cause Software defect

IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALBLotte E-commerce · 2022

Root cause DNS or network fault

IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgradedOpenAI · 2022

Root cause Operator error

IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errorsGitLab · 2022

Root cause Configuration change

IL-0146 New AKS clusters with Azure CNI in Central India lost tunnel connectivity because VMs began getting MAC addresses with a 06 prefix, which exposed a dormant bug in bridge-mode Azure CNIMicrosoft Azure · 2020

Root cause Software defect

IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutesGrafana Labs · 2020

Root cause Software defect

IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeoutsOmio · 2020

Root cause Software defect

IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 secondsAirbnb · 2019

Root cause Capacity exhaustion

IL-0057 Higher p99 latency after moving a reverse proxy to EKS, from outgoing connections delayed by a SNAT conntrack racePrezi · 2019

Root cause DNS or network fault

IL-0047 Pod-to-pod requests delayed by 5, 10 or 15 seconds because of DNS lookups lost to a conntrack raceMindTickle · 2019

Root cause DNS or network fault

IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memoryNU.nl · 2019

Root cause Configuration change