Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

Ignored early signal

초기 신호 무시 · early-signal-ignored

14 cases · 2017 to 2025

Root cause Software defect4 Capacity exhaustion3 Configuration change2 DNS or network fault2 Certificate or credential expiry1 Cloud provider outage1 unknown1

Component Kubernetes API server3 Admission webhook2 Configuration templating2 CoreDNS2

Symptom Service outage5 Control plane unresponsive3 High error rate3 OOMKilled3

Organisation Adevinta1 Airbnb1 Anthropic1 Barush Mendez1 Blue Matador

Cases

IL-0022 Three overlapping infrastructure bugs (context-window misrouting, TPU output corruption and an XLA top-k miscompilation) degraded Claude's responsesAnthropic · 2025
Root cause Software defect
IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six daysBarush Mendez · 2025
Root cause Software defect
Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS

View on the graph

Horizontal pod autoscaler2

Ingress controller2

Container memory limit2

Telemetry agent2

Cloud load balancer1

Cluster autoscaler1

Cluster dashboard1

conntrack table1

Container runtime1

Client-side DNS resolver and cache1

GKE control plane1

Model request routing1

Node root disk1

Cluster authentication provider1

Service discovery1

Deploys failing2

DNS resolution failures2

Monitoring outage2

Probe failures2

Load imbalance2

Autoscaler misbehaviour1

Scale-up failure1

Container crash loop1

Image pull failures1

Intermittent connection failures1

Sustained high node CPU1

Misrouted requests1

Response quality degradation1

Stale endpoint routing1

Throughput degradation1

Unauthorised access or a foreign workload1

1

IL-0083

Adevinta · 2023

Root cause DNS or network fault

IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rolloutGrafana Labs · 2022

Root cause Configuration change

IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutesDFDS · 2022

Root cause Certificate or credential expiry

IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two podsAirbnb · 2020

Root cause Software defect

IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entriesPreply · 2020

Root cause DNS or network fault

IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limitsBlue Matador · 2019

Root cause Capacity exhaustion

IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loopFullstaq · 2019

Root cause Capacity exhaustion

IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP orderSkyscanner · 2019

Root cause Software defect

IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPAloveholidays · 2019

Root cause Cloud provider outage

IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboardJW Player · 2019

Root cause Configuration change

IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probeNRE Labs · 2018

Root cause unknown

IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker downSaltside · 2017

Root cause Capacity exhaustion