Incident Library
Applied filters ComponentContainer memory limitSymptomOOMKilledClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases13 cases

13 to 13 of 13

  1. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019
    Symptoms

    Customers of the Zalando Fashion Store got a high rate of errors on all web product pages and outfit pages for more than an hour. The errors came from the main data aggregation layer service, which runs in one of Zalando's Kubernetes clusters.

    High error rateDNS resolution failuresOOMKilledMonitoring outage
    Trigger

    Calls from the aggregation layer to a single, not very important downstream service started timing out. The aggregation layer returned 404s for them, clients retried, and the resulting spike in requests became a spike in DNS queries to CoreDNS.

    Root cause
    Capacity exhaustion
    Component
    CoreDNSContainer memory limitClient-side DNS resolver and cache
    Contributing factor
    Unknown limitNo cachingRetry stormUnreviewed defaultsExcessive blast radiusMonitoring blind spot
    Detection

    The on-call engineer for the aggregation layer phoned the on-call Kubernetes engineer. Automatic paging did not happen because the cluster's monitoring and alerting ran inside the cluster and failed along with DNS. The Kubernetes engineer then saw that the CoreDNS pods were being OOM-killed.

    Remediation
    • Raised the CoreDNS memory requests and limits by hand from 100Mi to 2000Mi
    Action items
    • Introduce a more resilient DNS setup: CoreDNS fronted by dnsmasq on every node
    • Improve high-level alerting from outside the clusters, so that a cluster problem is known before or at the same time as applications are affected
    • Improve incident response processes and tooling so that any on-call engineer can escalate directly to other teams when automatic paging fails, for example through a chat bot
    Lessons

    The Kubernetes DNS setup was far from resilient enough: a single application could take down DNS for everything in the cluster. The blast radius of one application sending more DNS queries should be limited to one node at most, ideally to single pods. Known Kubernetes DNS problems, such as Linux kernel races in conntrack handling and the ndots 5 problem, were familiar to the team and partly mitigated, but their combined effect on the DNS infrastructure was worse than the team had understood. Monitoring that runs inside the cluster was not robust enough: when the cluster's DNS broke, it could not page anyone, and recovery took longer than it should have.

    View case
Check your cluster for this

These 13 cases involve Container memory limit and CoreDNS. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause3

Capacity exhaustion9 cases

Component
Container memory limit9CoreDNS3Node pool3Application database2Client-side DNS resolver and cache2kubelet2Prometheus2Telemetry agent2Horizontal pod autoscaler1Service discovery1
Contributing factor
Monitoring blind spot3Retry storm3Excessive blast radius2Unreviewed defaults2Unknown limit2No caching2No resource requests or limits2No headroom2Single point of failure2Automation amplification1Ignored early signal1Harmful manual action1No alert on limit1
Organisation
Backpackr1Blue Matador1Buildkite1DFDS1GitLab1Saifeddine Rajhi1Scaleway1UK Ministry of Justice1Zalando1
  1. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch
    Buildkite2026
  2. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down
    Backpackr2025
  3. IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgrade
    Saifeddine Rajhi2024
  4. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day
    DFDS2024
  5. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady
    Scaleway2024
  6. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady
    UK Ministry of Justice2023
  7. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary
    GitLab2022
  8. IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limits
    Blue Matador2019
  9. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019

Configuration change3 cases

Component
Container memory limit3In-memory cache1Configuration templating1kubelet1Persistent volumes and stateful sets1Prometheus1Service discovery1
Contributing factor
Unreviewed defaults2Unverified assumption1Excessive blast radius1Ignored early signal1Monitoring blind spot1No alert on limit1No health probes1No resource requests or limits1No headroom1Retry storm1Single point of failure1Unrepresentative staging1
Organisation
AdGuard VPN1Grafana Labs1NU.nl1
  1. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
  2. IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rollout
    Grafana Labs2022
  3. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory
    NU.nl2019

DNS or network fault1 case

Component
Container memory limitContainer registry and its credentials
Contributing factor
Harmful manual actionOverlapping failuresSingle point of failure
Organisation
GitHub
  1. IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registry
    GitHub2020