Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases18 cases

18 to 18 of 18

  1. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019
    Symptoms

    Customers of the Zalando Fashion Store got a high rate of errors on all web product pages and outfit pages for more than an hour. The errors came from the main data aggregation layer service, which runs in one of Zalando's Kubernetes clusters.

    High error rateDNS resolution failuresOOMKilledMonitoring outage
    Trigger

    Calls from the aggregation layer to a single, not very important downstream service started timing out. The aggregation layer returned 404s for them, clients retried, and the resulting spike in requests became a spike in DNS queries to CoreDNS.

    Root cause
    Capacity exhaustion
    Component
    CoreDNSContainer memory limitClient-side DNS resolver and cache
    Contributing factor
    Unknown limitNo cachingRetry stormUnreviewed defaultsExcessive blast radiusMonitoring blind spot
    Detection

    The on-call engineer for the aggregation layer phoned the on-call Kubernetes engineer. Automatic paging did not happen because the cluster's monitoring and alerting ran inside the cluster and failed along with DNS. The Kubernetes engineer then saw that the CoreDNS pods were being OOM-killed.

    Remediation
    • Raised the CoreDNS memory requests and limits by hand from 100Mi to 2000Mi
    Action items
    • Introduce a more resilient DNS setup: CoreDNS fronted by dnsmasq on every node
    • Improve high-level alerting from outside the clusters, so that a cluster problem is known before or at the same time as applications are affected
    • Improve incident response processes and tooling so that any on-call engineer can escalate directly to other teams when automatic paging fails, for example through a chat bot
    Lessons

    The Kubernetes DNS setup was far from resilient enough: a single application could take down DNS for everything in the cluster. The blast radius of one application sending more DNS queries should be limited to one node at most, ideally to single pods. Known Kubernetes DNS problems, such as Linux kernel races in conntrack handling and the ndots 5 problem, were familiar to the team and partly mitigated, but their combined effect on the DNS infrastructure was worse than the team had understood. Monitoring that runs inside the cluster was not robust enough: when the cluster's DNS broke, it could not page anyone, and recovery took longer than it should have.

    View case
Check your cluster for this

These 18 cases involve CoreDNS and Client-side DNS resolver and cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause5

Capacity exhaustion12 cases

Component
CoreDNS3Node pool3Client-side DNS resolver and cache2etcd2Kubernetes API server2Container memory limit2Prometheus2Telemetry agent2Admission webhook1Cloud controller manager1Cluster autoscaler1conntrack table1Container runtime1Application database1HAProxy1kube-proxy1kubelet1NAT gateway1Node root disk1Persistent volumes and stateful sets1Container registry and its credentials1VPC subnet or pod address range1VPC DNS resolver1
Contributing factor
Unknown limit12Unreviewed defaults6Monitoring blind spot4Excessive blast radius2No capacity planning2Retry storm2Single point of failure2Unverified assumption1Automation amplification1Limited debug access1Harmful manual action1No alert on limit1No caching1No runbook1No headroom1
Organisation
loveholidays2Civis Analytics1Datadog1DigitalOcean1Magic Pages1Onkar Raghunath Shelke1Palark1Saifeddine Rajhi1UK Ministry of Justice1unknown1Zalando1
  1. IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshots
    Onkar Raghunath Shelke2026
  2. IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodes
    DigitalOcean2026
  3. IL-0048 DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookups
    unknown2025
  4. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
  5. IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgrade
    Saifeddine Rajhi2024
  6. IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowed
    UK Ministry of Justice2024
  7. IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodes
    Palark2023
  8. IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resets
    loveholidays2020
  9. IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failing
    loveholidays2020
  10. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances
    Datadog2019
  11. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019
  12. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019

Dependency failure2 cases

Component
Admission webhook1Istio1Node operating system image1Container registry and its credentials1VPC DNS resolver1
Contributing factor
Unknown limit2Unreviewed defaults1
Organisation
Linode1Tetrate (unnamed U.S. federal customer)1
  1. IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiod
    Tetrate (unnamed U.S. federal customer)2025
  2. IL-0159 Docker Hub rate-limited Linode's IPv6 address space with 429 Too Many Requests, breaking image pulls for LKE and LKE-E clusters
    Linode2025

DNS or network fault1 case

Component
CoreDNSClient-side DNS resolver and cacheTelemetry agent
Contributing factor
Unverified assumptionUnreviewed defaultsIgnored early signalUnknown limit
Organisation
Adevinta
  1. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS
    Adevinta2023

Software defect2 cases

Component
Container CPU limit1kubelet1Linux kernel1Serverless function platform1
Contributing factor
Unknown limit2Unpatched known bug1Monitoring blind spot1
Organisation
Amazon Web Services1Omio1
  1. IL-0151 A latent defect in the AWS Lambda Frontend, set off when one us-east-1 cell scaled past a size never reached before, raised errors for Lambda and for services that depend on it, including new EKS cluster provisioning
    Amazon Web Services2023
  2. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020

Certificate or credential expiry1 case

Component
Kubernetes API serverkubelet
Contributing factor
Unknown limitNo alert on limit
Organisation
dbhang
  1. IL-0184 JupyterHub's KubeSpawner could not create containers and kubectl was refused because the API server and other internal certificates of a kubeadm cluster had expired
    dbhang2021