Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodes
    DigitalOcean2026
    Symptoms

    From 20:31 UTC on July 9, 2026, DigitalOcean reported a problem with Kubernetes deployments in its NYC1 region. Application workloads could see intermittent DNS failures and NodeNotReady events.

    DNS resolution failuresIntermittent timeoutsNode NotReady
    Root cause
    Capacity exhaustion
    Component
    CoreDNSNode pool
    Contributing factor
    Unknown limit
    Lessons

    Cluster DNS is sensitive to latency, and CoreDNS on shared-CPU nodes can time out intermittently. DigitalOcean's advice is to place CoreDNS on dedicated-CPU node pools and to run enough CoreDNS replicas.

    View case
  2. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady
    UK Ministry of Justice2023
    Symptoms

    At 14:05 on 25 July 2023, a high-priority PagerDuty alert from Pingdom reported that the "Prometheus - live healthcheck" was DOWN. Prometheus on the live cluster was unavailable, and the Cloud Platform lost its monitoring for a while.

    Monitoring outageOOMKilledNode NotReady
    Trigger

    The growing number of namespaces and Prometheus rules raised Prometheus' memory needs beyond what its node instance size could hold.

    Root cause
    Capacity exhaustion
    Component
    PrometheusNode poolContainer memory limit
    Contributing factor
    No alert on limit
    Detection

    A Pingdom health check on Prometheus failed and raised a high-priority PagerDuty alert.

    Remediation
    • drained the monitoring node, moving Prometheus to another monitoring node
    • changed the node instance size to r6i.4xlarge
    • raised the Prometheus memory limit to 100Gi
    • set the Prometheus container limits to 12 CPU cores and 110Gi of memory
    Action items
    • Add an alert on node memory usage and on a pod using up most of a node's memory (issue #4538)
    View case
Check your cluster for this

These 2 cases involve Node pool and CoreDNS. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion2 cases

Component
Node pool2CoreDNS1Container memory limit1Prometheus1
Contributing factor
Unknown limit1No alert on limit1
Organisation
DigitalOcean1UK Ministry of Justice1
  1. IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodes
    DigitalOcean2026
  2. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady
    UK Ministry of Justice2023