Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

etcd

etcd · etcd

15 cases · 2017 to 2026

Root cause Capacity exhaustion4 Software defect4 Configuration change2 Certificate or credential expiry1 Dependency failure1 Hardware failure1 unknown1 Breaking change in an upgrade1

Contributing factor Unreviewed defaults7 Monitoring blind spot6 Single point of failure5 Harmful manual action4

Symptom Control plane unresponsive9 High error rate6 Service outage5 Control plane lag3

Organisation Civis Analytics1 ckck8031 DigitalOcean1 Fullstaq1 GitHub

Cases, by severity

IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limitMarcin Cuber · 2024 · sev1
Root cause Software defect
IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returnedNebius · 2026
Root cause Dependency failure
IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loads

View on the graph

Ignored early signal2

Limited debug access2

No alert on limit2

No safeguard on destructive change2

Overlapping failures2

Unverified assumption1

Automation amplification1

Untested backup1

Excessive blast radius1

Configuration not in code1

No capacity planning1

Ineffective rollback1

Unrepresentative staging1

DNS resolution failures3

Monitoring outage3

Stale endpoint routing2

Container crash loop1

Database overload1

Deploys failing1

Intermittent timeouts1

Pods stuck in ContainerCreating1

Probe failures1

Resources deleted1

Stale or inconsistent API reads1

1

GitHub · 2026

Root cause Configuration change

IL-0183 kubeadm's one-year leaf certificates expired, so kube-apiserver could not reach etcd, restarted in a loop and left the cluster unreachableckck803 · 2026

Root cause Certificate or credential expiry

IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API serverRender · 2024

Root cause Capacity exhaustion

IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1DigitalOcean · 2024

Root cause Configuration change

IL-0185 A disk failure made the first control plane node of a Kubespray cluster read-only, and recovery needed the Kubespray inventory reordered and a new control plane node joinedHans H. Kim · 2023

Root cause Hardware failure

IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodesPalark · 2023

Root cause Capacity exhaustion

IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutesReddit · 2023

Root cause Breaking change in an upgrade

IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutesGrafana Labs · 2020

Root cause Software defect

IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loopFullstaq · 2019

Root cause Capacity exhaustion

IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the APICivis Analytics · 2019

Root cause Capacity exhaustion

IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probeNRE Labs · 2018

Root cause unknown

IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its stateNordstrom · 2017

Root cause Software defect

IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outageMonzo · 2017

Root cause Software defect