Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

Kubernetes API server

쿠버네티스 API 서버 · kube-apiserver

28 cases · 2017 to 2026

Root cause Capacity exhaustion9 Configuration change6 Software defect5 Certificate or credential expiry3 Cloud provider outage1 Credential exposure1 DNS or network fault1 unknown1 Breaking change in an upgrade1

Contributing factor Monitoring blind spot9 Unreviewed defaults8 Single point of failure7 Excessive blast radius5

Symptom Control plane unresponsive18 Service outage9 Control plane lag8 High error rate8

Organisation Zalando3 DigitalOcean2 OpenAI2 Palark2 Aqua Security

Cases, by severity

IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limitMarcin Cuber · 2024 · sev1
Root cause Software defect
IL-0069 Attackers stole a high-privilege service account token from a production Kubernetes cluster and used it to reach a cryptocurrency exchange's financial systemsUnnamed cryptocurrency exchange · 2026
Root cause Credential exposure
kubeadm's one-year leaf certificates expired, so kube-apiserver could not reach etcd, restarted in a loop and left the cluster unreachable

View on the graph

Limited debug access5

Harmful manual action4

Automation amplification3

Ignored early signal3

Unverified assumption2

No alert on limit2

No capacity planning2

No staged rollout2

Overlapping failures2

Unrepresentative staging2

Untested backup1

Configuration not in code1

Change with a delayed effect1

Cache-masked failure1

Unpatched known bug1

No configuration tests1

No health probes1

No safeguard on destructive change1

Ineffective rollback1

DNS resolution failures4

Deploys failing3

Monitoring outage3

Credentials unavailable2

Intermittent connection failures2

Resources deleted2

Stale or inconsistent API reads2

Stale endpoint routing2

Unauthorised access or a foreign workload2

Container crash loop1

Database overload1

Node recreation loop1

Pods stuck in ContainerCreating1

Probe failures1

1

Civis Analytics1

Unnamed cryptocurrency exchange1

IL-0183

ckck803 · 2026

Root cause Certificate or credential expiry

IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six daysBarush Mendez · 2025

Root cause Software defect

IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API serverRender · 2024

Root cause Capacity exhaustion

IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI servicesOpenAI · 2024

Root cause Configuration change

IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clustersOpenAI · 2024

Root cause Configuration change

IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPUNamespace · 2024

Root cause Cloud provider outage

IL-0154 Line cards on an SFO3 core router rebooted together and halved backbone capacity, and DOKS nodes whose guest networking did not recover stayed NotReady with degraded API servers until all affected nodes in the SFO3 DOKS fleet were recycledDigitalOcean · 2024

Root cause DNS or network fault

IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PARScaleway · 2024

Root cause Capacity exhaustion

IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1DigitalOcean · 2024

Root cause Configuration change

IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agentsPalark · 2023

Root cause Capacity exhaustion

IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023

Root cause Capacity exhaustion

IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSetAqua Security · 2023

Root cause Configuration change

IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodesPalark · 2023

Root cause Capacity exhaustion

IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutesReddit · 2023

Root cause Breaking change in an upgrade

IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effectWebhook Relay · 2022

Root cause Certificate or credential expiry

IL-0184 JupyterHub's KubeSpawner could not create containers and kubectl was refused because the API server and other internal certificates of a kubeadm cluster had expireddbhang · 2021

Root cause Certificate or credential expiry

IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time outDFDS · 2020

Root cause Configuration change

IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every nodeJetstack · 2019

Root cause Configuration change

IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limitZalando · 2019

Root cause Capacity exhaustion

IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loopFullstaq · 2019

Root cause Capacity exhaustion

IL-0030 Most nodes of a cluster went NotReady together for 15 minutes while the API server sat behind an AWS ELBZalando · 2019

Root cause Software defect

IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the APICivis Analytics · 2019

Root cause Capacity exhaustion

IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifestZalando · 2018

Root cause Capacity exhaustion

IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probeNRE Labs · 2018

Root cause unknown

IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its stateNordstrom · 2017

Root cause Software defect

IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outageMonzo · 2017

Root cause Software defect