Incident Library
Applied filters ComponentetcdSymptomControl plane unresponsiveClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases9 cases

9 to 9 of 9

  1. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe
    NRE Labs2018
    Symptoms

    The daily NRE Labs statistics showed lesson activity collapsing, and kubectl could not reach the cluster: `The connection to the server 1.2.3.4:6443 was refused - did you specify the right host or port?`

    Control plane unresponsiveProbe failuresService outage
    Trigger

    Unknown. The liveness probe that kubeadm writes into the etcd static pod manifest (an `etcdctl get foo` against 127.0.0.1:2379, failureThreshold 8, timeoutSeconds 15) started failing. The author first suspected GCP maintenance the night before, but could not reproduce the probe failure by hand and does not know why it began.

    Root cause
    unknown
    Component
    etcdkubeletKubernetes API server
    Contributing factor
    Single point of failureUnreviewed defaultsHarmful manual actionIgnored early signalMonitoring blind spot
    Detection

    The author saw lesson activity drop in the daily NRE Labs statistics during his morning routine.

    Remediation
    • removed the liveness probe section from the kubeadm-generated etcd static pod manifest and applied it between etcd restarts
    Action items
    • Add monitoring, especially for the single-node master
    • Run a basic "down for maintenance" page in Google Cloud Functions that points to Twitter for updates, to redirect to during outages
    Lessons

    Because all of the infrastructure is managed as code, the cluster could be rebuilt quickly, but destroying it before finding the cause also destroyed the evidence. Monitoring matters, especially with a single-node master. Read the restart counts in kubectl output, not just the Running status. Remember the fundamentals instead of expecting the issue to be more complicated than it is, and do not let the urgency of an outage push you into hasty decisions. Put up a maintenance page instead of leaving the site failing halfway.

    View case
Check your cluster for this

These 9 cases involve etcd and Kubernetes API server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause6

Dependency failure1 case

Component
Cloud block storageData center power and coolingetcdInfiniBand fabric
Contributing factor
Monitoring blind spotNo alert on limitNo runbookOverlapping failuresSingle point of failure
Organisation
Nebius
  1. IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returned
    Nebius2026

Certificate or credential expiry1 case

Component
CalicoetcdKubernetes API serverkubelet
Contributing factor
Unreviewed defaults
Organisation
ckck803
  1. IL-0183 kubeadm's one-year leaf certificates expired, so kube-apiserver could not reach etcd, restarted in a loop and left the cluster unreachable
    ckck8032026

Capacity exhaustion4 cases

Component
etcd4Kubernetes API server4Telemetry agent2CoreDNS1Cluster authentication provider1
Contributing factor
Unreviewed defaults3Unknown limit2Harmful manual action2Unverified assumption1Ignored early signal1Limited debug access1Monitoring blind spot1No capacity planning1Retry storm1
Organisation
Civis Analytics1Fullstaq1Palark1Render1
  1. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024
  2. IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodes
    Palark2023
  3. IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loop
    Fullstaq2019
  4. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019

Software defect1 case

Component
etcdKubernetes API serverPolicy engine
Contributing factor
Limited debug accessNo alert on limit
Organisation
Marcin Cuber
  1. IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limit
    Marcin Cuber2024

Configuration change1 case

Component
etcdKubernetes API server
Contributing factor
Automation amplificationExcessive blast radiusNo safeguard on destructive change
Organisation
DigitalOcean
  1. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1
    DigitalOcean2024

unknown1 case

Component
etcdKubernetes API serverkubelet
Contributing factor
Unreviewed defaultsIgnored early signalHarmful manual actionMonitoring blind spotSingle point of failure
Organisation
NRE Labs
  1. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe
    NRE Labs2018