Incident Library
Applied filters ComponentKubernetes API serverSymptomDeploys failingClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024
    Symptoms

    In January 2022 an etcd memory spike overloaded the Kubernetes control plane of one of Render's clusters in Frankfurt. At first only operations that need the control plane failed: new deploys and cron job runs.

    Control plane unresponsiveDeploys failingDNS resolution failuresHigh error rate
    Trigger

    A memory spike in etcd overloaded the cluster's control plane.

    Root cause
    Capacity exhaustion
    Component
    etcdCoreDNSKubernetes API server
    Contributing factor
    Unreviewed defaultsHarmful manual action
    Action items
    • Run CoreDNS on data plane nodes instead of control plane nodes
    • Run etcd on dedicated nodes
    Lessons

    CoreDNS can keep answering from its in-memory cache while the API server is down, but it cannot initialise without it. With CoreDNS on the control plane nodes, a reboot or restart during a control plane outage turned a failure of deploys and cron jobs into a failure of requests to services that were already running. The core problem was that the data plane was not isolated enough from the control plane: even without the reboot, the etcd memory spike alone could have forced CoreDNS to restart.

    View case
  2. IL-0184 JupyterHub's KubeSpawner could not create containers and kubectl was refused because the API server and other internal certificates of a kubeadm cluster had expired
    dbhang2021
    Symptoms

    KubeSpawner, which starts JupyterHub user containers on Kubernetes, failed to create containers with an error the author had never seen, saying an SSL certificate had expired.

    Control plane unresponsiveDeploys failing
    Root cause
    Certificate or credential expiry
    Component
    Kubernetes API serverkubelet
    Contributing factor
    Unknown limitNo alert on limit
    Detection

    KubeSpawner failed to create a container and logged an SSL certificate verification error.

    Remediation
    • checked certificate expiry with kubeadm alpha certs check-expiration
    • moved the old certificates and /etc/kubernetes/*.conf files to a backup directory
    • regenerated all certificates with kubeadm init phase certs all
    • regenerated the kubeconfig files with kubeadm init phase kubeconfig all
    • copied /etc/kubernetes/admin.conf to ~/.kube/config and restarted kubelet
    • restarted the kube-system containers with docker restart so they picked up the new credentials
    Lessons

    kubeadm clusters run on certificates that expire, and when the API server certificate expires kubectl cannot be used to fix it, so renewal has to happen on the control plane node with kubeadm. After renewal, the kubeconfig files and the running control plane containers must also pick up the new certificates, or kubectl returns Unauthorized.

    View case
  3. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
    DFDS2020
    Symptoms

    At 14:41 CET on 4 November 2020 users reported timeouts when deploying to Hellman, the production Kubernetes cluster. The timeouts were confirmed at 15:04.

    Deploys failingControl plane lag
    Trigger

    During an Alcide workshop that began at 13:00 CET, an Alcide admission controller was applied to the production cluster at 13:13 in the belief that it would not disrupt anything.

    Root cause
    Configuration change
    Component
    Admission webhookKubernetes API server
    Contributing factor
    No configuration testsUnverified assumptionMonitoring blind spot
    Detection

    Users reported timeouts when deploying to the Hellman cluster at 14:41 CET; the timeouts were verified at 15:04.

    Remediation
    • toggled the Alcide admission controller to verify that it caused the timeouts
    • permanently deleted the Alcide admission controller
    Action items
    • Monitor Kubernetes deployments for timeouts
    • Monitor 5xx status codes from the Kubernetes API server
    Lessons

    New components are applied in sandbox clusters, not in production. The team needs enough Alcide knowledge to maintain the component until it is fully deprecated.

    View case
Check your cluster for this

These 3 cases involve Kubernetes API server and Admission webhook. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause3

Capacity exhaustion1 case

Component
CoreDNSetcdKubernetes API server
Contributing factor
Unreviewed defaultsHarmful manual action
Organisation
Render
  1. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024

Certificate or credential expiry1 case

Component
Kubernetes API serverkubelet
Contributing factor
Unknown limitNo alert on limit
Organisation
dbhang
  1. IL-0184 JupyterHub's KubeSpawner could not create containers and kubectl was refused because the API server and other internal certificates of a kubeadm cluster had expired
    dbhang2021

Configuration change1 case

Component
Admission webhookKubernetes API server
Contributing factor
Unverified assumptionMonitoring blind spotNo configuration tests
Organisation
DFDS
  1. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
    DFDS2020