Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Control plane unresponsive

컨트롤 플레인 무응답 · control-plane-unresponsive

22 cases · 2018 to 2026

Root cause Capacity exhaustion7 Configuration change4 Certificate or credential expiry3 Dependency failure2 Software defect2 Cloud provider outage1 Faulty deployment1 DNS or network fault1 unknown1

Component Kubernetes API server18 etcd9 kubelet5 Telemetry agent4

Contributing factor Unreviewed defaults6 Monitoring blind spot6 Single point of failure6 Excessive blast radius5

Organisation Linode3 DigitalOcean2 OpenAI2 Palark2 Barush Mendez1

Cases

IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returnedNebius · 2026
Root cause Dependency failure
IL-0160 A repaired legacy GPU monitoring script sent a firmware query that triggers a race condition in the GPU System Processor of NVIDIA RTX 4000 Ada GPUs, locking up GPU Linodes and LKE worker nodes in several regionsLinode · 2026
Root cause Faulty deployment
kubeadm's one-year leaf certificates expired, so kube-apiserver could not reach etcd, restarted in a loop and left the cluster unreachable

View on the graph

Application database2

GKE control plane2

Admission webhook1

AKS control plane1

Cloud block storage1

Data center power and cooling1

Client-side DNS resolver and cache1

Authoritative DNS servers and records1

InfiniBand fabric1

Ingress controller1

Container memory limit1

Cluster authentication provider1

RBAC and service accounts1

Limited debug access5

Ignored early signal3

Harmful manual action3

No alert on limit3

Unverified assumption2

Automation amplification2

Change with a delayed effect1

Cache-masked failure1

No capacity planning1

No health probes1

No safeguard on destructive change1

No staged rollout1

Overlapping failures1

Missed in review1

Unrepresentative staging1

Civis Analytics1

Microsoft Azure1

IL-0183

ckck803 · 2026

Root cause Certificate or credential expiry

IL-0162 LKE users could not reach their cluster control planes or the LKE Dashboard because an earlier database update made the internal LKE DNS service hangLinode · 2025

Root cause Dependency failure

IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six daysBarush Mendez · 2025

Root cause Software defect

IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API serverRender · 2024

Root cause Capacity exhaustion

IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI servicesOpenAI · 2024

Root cause Configuration change

IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clustersOpenAI · 2024

Root cause Configuration change

IL-0154 Line cards on an SFO3 core router rebooted together and halved backbone capacity, and DOKS nodes whose guest networking did not recover stayed NotReady with degraded API servers until all affected nodes in the SFO3 DOKS fleet were recycledDigitalOcean · 2024

Root cause DNS or network fault

IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limitMarcin Cuber · 2024 · sev1

Root cause Software defect

IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1DigitalOcean · 2024

Root cause Configuration change

IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia SoutheastMicrosoft Azure · 2023

Root cause Cloud provider outage

IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agentsPalark · 2023

Root cause Capacity exhaustion

IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023

Root cause Capacity exhaustion

IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodesPalark · 2023

Root cause Capacity exhaustion

IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effectWebhook Relay · 2022

Root cause Certificate or credential expiry

IL-0184 JupyterHub's KubeSpawner could not create containers and kubectl was refused because the API server and other internal certificates of a kubeadm cluster had expireddbhang · 2021

Root cause Certificate or credential expiry

IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every nodeJetstack · 2019

Root cause Configuration change

IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loopFullstaq · 2019

Root cause Capacity exhaustion

IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the APICivis Analytics · 2019

Root cause Capacity exhaustion

IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifestZalando · 2018

Root cause Capacity exhaustion

IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probeNRE Labs · 2018

Root cause unknown