Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

Unknown limit

한계 미인지 · limit-unknown

18 cases · 2019 to 2026

Root cause Capacity exhaustion12 Dependency failure2 Software defect2 Certificate or credential expiry1 DNS or network fault1

Component CoreDNS4 Client-side DNS resolver and cache3 Kubernetes API server3 kubelet3

Symptom High error rate6 DNS resolution failures5 High latency4 Probe failures4

Organisation loveholidays2 Adevinta1 Amazon Web Services1 Civis Analytics1

Cases

IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshotsOnkar Raghunath Shelke · 2026
Root cause Capacity exhaustion
IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodesDigitalOcean · 2026
Root cause Capacity exhaustion
IL-0048 DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookups

View on the graph

Telemetry agent3

Admission webhook2

Container memory limit2

Container registry and its credentials2

VPC DNS resolver2

Cloud controller manager1

Cluster autoscaler1

conntrack table1

Container runtime1

Container CPU limit1

Application database1

Node root disk1

Node operating system image1

Persistent volumes and stateful sets1

Serverless function platform1

VPC subnet or pod address range1

Control plane unresponsive3

Intermittent timeouts3

Monitoring outage3

Image pull failures2

Intermittent connection failures2

Service outage2

Scale-up failure1

Cloud API rate limit exhausted1

Connection resets1

Control plane lag1

CPU throttling1

Container crash loop1

Credentials unavailable1

Database overload1

Deploys failing1

IP address exhaustion1

Node disk full1

Onkar Raghunath Shelke1

Saifeddine Rajhi1

Tetrate (unnamed U.S. federal customer)1

UK Ministry of Justice1

unknown · 2025

Root cause Capacity exhaustion

IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hoursMagic Pages · 2025

Root cause Capacity exhaustion

IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiodTetrate (unnamed U.S. federal customer) · 2025 · sev1

Root cause Dependency failure

IL-0159 Docker Hub rate-limited Linode's IPv6 address space with 429 Too Many Requests, breaking image pulls for LKE and LKE-E clustersLinode · 2025

Root cause Dependency failure

IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024

Root cause Capacity exhaustion

IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowedUK Ministry of Justice · 2024

Root cause Capacity exhaustion

IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNSAdevinta · 2023

Root cause DNS or network fault

IL-0151 A latent defect in the AWS Lambda Frontend, set off when one us-east-1 cell scaled past a size never reached before, raised errors for Lambda and for services that depend on it, including new EKS cluster provisioningAmazon Web Services · 2023

Root cause Software defect

IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodesPalark · 2023

Root cause Capacity exhaustion

IL-0184 JupyterHub's KubeSpawner could not create containers and kubectl was refused because the API server and other internal certificates of a kubeadm cluster had expireddbhang · 2021

Root cause Certificate or credential expiry

IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeoutsOmio · 2020

Root cause Software defect

IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020

Root cause Capacity exhaustion

IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failingloveholidays · 2020

Root cause Capacity exhaustion

IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instancesDatadog · 2019

Root cause Capacity exhaustion

IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the APICivis Analytics · 2019

Root cause Capacity exhaustion

IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019

Root cause Capacity exhaustion