Incident Library
Applied filters SymptomControl plane unresponsiveClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases22 cases

22 to 22 of 22

  1. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe
    NRE Labs2018
    Symptoms

    The daily NRE Labs statistics showed lesson activity collapsing, and kubectl could not reach the cluster: `The connection to the server 1.2.3.4:6443 was refused - did you specify the right host or port?`

    Control plane unresponsiveProbe failuresService outage
    Trigger

    Unknown. The liveness probe that kubeadm writes into the etcd static pod manifest (an `etcdctl get foo` against 127.0.0.1:2379, failureThreshold 8, timeoutSeconds 15) started failing. The author first suspected GCP maintenance the night before, but could not reproduce the probe failure by hand and does not know why it began.

    Root cause
    unknown
    Component
    etcdkubeletKubernetes API server
    Contributing factor
    Single point of failureUnreviewed defaultsHarmful manual actionIgnored early signalMonitoring blind spot
    Detection

    The author saw lesson activity drop in the daily NRE Labs statistics during his morning routine.

    Remediation
    • removed the liveness probe section from the kubeadm-generated etcd static pod manifest and applied it between etcd restarts
    Action items
    • Add monitoring, especially for the single-node master
    • Run a basic "down for maintenance" page in Google Cloud Functions that points to Twitter for updates, to redirect to during outages
    Lessons

    Because all of the infrastructure is managed as code, the cluster could be rebuilt quickly, but destroying it before finding the cause also destroyed the evidence. Monitoring matters, especially with a single-node master. Read the restart counts in kubectl output, not just the Running status. Remember the fundamentals instead of expecting the issue to be more complicated than it is, and do not let the urgency of an outage push you into hasty decisions. Put up a maintenance page instead of leaving the site failing halfway.

    View case
Check your cluster for this

These 22 cases involve Kubernetes API server and etcd. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause9

Dependency failure2 cases

Component
Cloud block storage1Application database1Data center power and cooling1Authoritative DNS servers and records1etcd1InfiniBand fabric1
Contributing factor
Monitoring blind spot1No alert on limit1No runbook1Overlapping failures1Single point of failure1
Organisation
Linode1Nebius1
  1. IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returned
    Nebius2026
  2. IL-0162 LKE users could not reach their cluster control planes or the LKE Dashboard because an earlier database update made the internal LKE DNS service hang
    Linode2025

Faulty deployment1 case

Component
GPU driverTelemetry agent
Contributing factor
Unverified assumptionMissed in review
Organisation
Linode
  1. IL-0160 A repaired legacy GPU monitoring script sent a firmware query that triggers a race condition in the GPU System Processor of NVIDIA RTX 4000 Ada GPUs, locking up GPU Linodes and LKE worker nodes in several regions
    Linode2026

Certificate or credential expiry3 cases

Component
Kubernetes API server3kubelet2Calico1etcd1GKE control plane1Node pool1RBAC and service accounts1
Contributing factor
Unreviewed defaults1Unknown limit1Limited debug access1No alert on limit1Single point of failure1
Organisation
ckck8031dbhang1Webhook Relay1
  1. IL-0183 kubeadm's one-year leaf certificates expired, so kube-apiserver could not reach etcd, restarted in a loop and left the cluster unreachable
    ckck8032026
  2. IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effect
    Webhook Relay2022
  3. IL-0184 JupyterHub's KubeSpawner could not create containers and kubectl was refused because the API server and other internal certificates of a kubeadm cluster had expired
    dbhang2021

Software defect2 cases

Component
Kubernetes API server2etcd1GPU driver1Linux kernel1Policy engine1
Contributing factor
Change with a delayed effect1Ignored early signal1Limited debug access1Monitoring blind spot1No alert on limit1Single point of failure1
Organisation
Barush Mendez1Marcin Cuber1
  1. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days
    Barush Mendez2025
  2. IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limit
    Marcin Cuber2024

Capacity exhaustion7 cases

Component
Kubernetes API server7etcd4Telemetry agent2Cilium1CoreDNS1Ingress controller1Container memory limit1Cluster authentication provider1
Contributing factor
Unreviewed defaults4Unknown limit2Harmful manual action2Unverified assumption1Excessive blast radius1Ignored early signal1Limited debug access1Monitoring blind spot1No capacity planning1Retry storm1Single point of failure1
Organisation
Palark2Civis Analytics1Fullstaq1Linode1Render1Zalando1
  1. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024
  2. IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agents
    Palark2023
  3. IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraints
    Linode2023
  4. IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodes
    Palark2023
  5. IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loop
    Fullstaq2019
  6. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019
  7. IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifest
    Zalando2018

Configuration change4 cases

Component
Kubernetes API server4Admission webhook1Client-side DNS resolver and cache1etcd1GKE control plane1kubelet1Telemetry agent1
Contributing factor
Excessive blast radius3Automation amplification2Limited debug access2Monitoring blind spot2Cache-masked failure1No health probes1No safeguard on destructive change1No staged rollout1Unrepresentative staging1
Organisation
OpenAI2DigitalOcean1Jetstack1
  1. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  2. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024
  3. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1
    DigitalOcean2024
  4. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019

DNS or network fault1 case

Component
Kubernetes API serverkubelet
Contributing factor
–
Organisation
DigitalOcean
  1. IL-0154 Line cards on an SFO3 core router rebooted together and halved backbone capacity, and DOKS nodes whose guest networking did not recover stayed NotReady with degraded API servers until all affected nodes in the SFO3 DOKS fleet were recycled
    DigitalOcean2024

Cloud provider outage1 case

Component
AKS control planeApplication database
Contributing factor
Excessive blast radiusSingle point of failure
Organisation
Microsoft Azure
  1. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast
    Microsoft Azure2023

unknown1 case

Component
etcdKubernetes API serverkubelet
Contributing factor
Unreviewed defaultsIgnored early signalHarmful manual actionMonitoring blind spotSingle point of failure
Organisation
NRE Labs
  1. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe
    NRE Labs2018