Incident Library
Applied filters SymptomDeploys failingFactorHarmful manual actionOrganisationRenderClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024
    Symptoms

    In January 2022 an etcd memory spike overloaded the Kubernetes control plane of one of Render's clusters in Frankfurt. At first only operations that need the control plane failed: new deploys and cron job runs.

    Control plane unresponsiveDeploys failingDNS resolution failuresHigh error rate
    Trigger

    A memory spike in etcd overloaded the cluster's control plane.

    Root cause
    Capacity exhaustion
    Component
    etcdCoreDNSKubernetes API server
    Contributing factor
    Unreviewed defaultsHarmful manual action
    Action items
    • Run CoreDNS on data plane nodes instead of control plane nodes
    • Run etcd on dedicated nodes
    Lessons

    CoreDNS can keep answering from its in-memory cache while the API server is down, but it cannot initialise without it. With CoreDNS on the control plane nodes, a reboot or restart during a control plane outage turned a failure of deploys and cron jobs into a failure of requests to services that were already running. The core problem was that the data plane was not isolated enough from the control plane: even without the reboot, the etcd memory spike alone could have forced CoreDNS to restart.

    View case
Check your cluster for this

This case involves CoreDNS and etcd. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
CoreDNSetcdKubernetes API server
Contributing factor
Unreviewed defaultsHarmful manual action
Organisation
Render
  1. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024