Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

Harmful manual action

수동 조치 악화 · manual-action-worsened

13 cases · 2017 to 2026

Root cause Capacity exhaustion7 Software defect2 unknown2 Configuration change1 DNS or network fault1

Component etcd4 Kubernetes API server4 Container runtime2 CoreDNS2

Symptom Service outage6 Control plane unresponsive3 Monitoring outage3 Container crash loop2

Organisation DFDS2 Argo CD users1 Datadog1 Fullstaq1 Fyra Labs1

Cases

IL-0058 Argo CD deleted namespaces shared by several applications, leaving about 97 applications stuck Missing and OutOfSyncArgo CD users · 2026
Root cause Software defect
IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletionNeon · 2025
Root cause Capacity exhaustion
IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server

View on the graph

Application database2

Container memory limit2

Container registry and its credentials2

Telemetry agent2

Admission webhook1

Cluster autoscaler1

Horizontal pod autoscaler1

Ingress controller1

Node launch template1

Multus meta-CNI1

Cluster authentication provider1

Pod scheduling and placement1

VPC subnet or pod address range1

High error rate2

Image pull failures2

Resources deleted2

Autoscaler misbehaviour1

Deploys failing1

DNS resolution failures1

Intermittent connection failures1

IP address exhaustion1

Sustained high node CPU1

Probe failures1

Stale endpoint routing1

UK Ministry of Justice1

Render · 2024

Root cause Capacity exhaustion

IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation이언철 · 2024

Root cause Capacity exhaustion

IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024

Root cause Capacity exhaustion

IL-0087 After a cold reboot of RKE2 nodes, pods could not get IP addresses because Multus could not reach the Kubernetes API, and further reboots spread the failure to every nodeFyra Labs · 2023

Root cause unknown

IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1

Root cause Configuration change

IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memoryUK Ministry of Justice · 2020

Root cause Capacity exhaustion

IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registryGitHub · 2020

Root cause DNS or network fault

IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loopFullstaq · 2019

Root cause Capacity exhaustion

IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instancesDatadog · 2019

Root cause Capacity exhaustion

IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probeNRE Labs · 2018

Root cause unknown

IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outageMonzo · 2017

Root cause Software defect