Incident Library
Applied filters ComponentKubernetes API serverSymptomHigh error rateFactorUnreviewed defaultsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024
    Symptoms

    In January 2022 an etcd memory spike overloaded the Kubernetes control plane of one of Render's clusters in Frankfurt. At first only operations that need the control plane failed: new deploys and cron job runs.

    Control plane unresponsiveDeploys failingDNS resolution failuresHigh error rate
    Trigger

    A memory spike in etcd overloaded the cluster's control plane.

    Root cause
    Capacity exhaustion
    Component
    etcdCoreDNSKubernetes API server
    Contributing factor
    Unreviewed defaultsHarmful manual action
    Action items
    • Run CoreDNS on data plane nodes instead of control plane nodes
    • Run etcd on dedicated nodes
    Lessons

    CoreDNS can keep answering from its in-memory cache while the API server is down, but it cannot initialise without it. With CoreDNS on the control plane nodes, a reboot or restart during a control plane outage turned a failure of deploys and cron jobs into a failure of requests to services that were already running. The core problem was that the data plane was not isolated enough from the control plane: even without the reboot, the etcd memory spike alone could have forced CoreDNS to restart.

    View case
  2. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
    Symptoms

    Builds on Zalando's in-house Continuous Delivery Platform (CDP), which runs CI/CD on Kubernetes, timed out and failed for almost an hour on 11 June 2019. The Kubernetes builder pods could not get AWS IAM credentials.

    High error rateCredentials unavailableControl plane lag
    Trigger

    An unlucky rescaling of the CDP Kubernetes cluster left only one node available for builder pods. Builder pods were created and deleted on that node so quickly that kubelet fell behind.

    Root cause
    Capacity exhaustion
    Component
    kubeletCredential broker for podsKubernetes API serverCI system
    Contributing factor
    Unreviewed defaultsSingle point of failure
    Remediation
    • Scaled the CDP cluster up by hand so builder pods spread over more than one node
    Action items
    • Raise the kubelet QPS limit from the default 5 to 50 in the global cluster configuration
    View case
Check your cluster for this

These 2 cases involve Kubernetes API server and CI system. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion2 cases

Component
Kubernetes API server2CI system1CoreDNS1Credential broker for pods1etcd1kubelet1
Contributing factor
Unreviewed defaults2Harmful manual action1Single point of failure1
Organisation
Render1Zalando1
  1. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024
  2. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019