Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases16 cases

16 to 16 of 16

  1. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019
    Symptoms

    Customers of the Zalando Fashion Store got a high rate of errors on all web product pages and outfit pages for more than an hour. The errors came from the main data aggregation layer service, which runs in one of Zalando's Kubernetes clusters.

    High error rateDNS resolution failuresOOMKilledMonitoring outage
    Trigger

    Calls from the aggregation layer to a single, not very important downstream service started timing out. The aggregation layer returned 404s for them, clients retried, and the resulting spike in requests became a spike in DNS queries to CoreDNS.

    Root cause
    Capacity exhaustion
    Component
    CoreDNSContainer memory limitClient-side DNS resolver and cache
    Contributing factor
    Unknown limitNo cachingRetry stormUnreviewed defaultsExcessive blast radiusMonitoring blind spot
    Detection

    The on-call engineer for the aggregation layer phoned the on-call Kubernetes engineer. Automatic paging did not happen because the cluster's monitoring and alerting ran inside the cluster and failed along with DNS. The Kubernetes engineer then saw that the CoreDNS pods were being OOM-killed.

    Remediation
    • Raised the CoreDNS memory requests and limits by hand from 100Mi to 2000Mi
    Action items
    • Introduce a more resilient DNS setup: CoreDNS fronted by dnsmasq on every node
    • Improve high-level alerting from outside the clusters, so that a cluster problem is known before or at the same time as applications are affected
    • Improve incident response processes and tooling so that any on-call engineer can escalate directly to other teams when automatic paging fails, for example through a chat bot
    Lessons

    The Kubernetes DNS setup was far from resilient enough: a single application could take down DNS for everything in the cluster. The blast radius of one application sending more DNS queries should be limited to one node at most, ideally to single pods. Known Kubernetes DNS problems, such as Linux kernel races in conntrack handling and the ndots 5 problem, were familiar to the team and partly mitigated, but their combined effect on the DNS infrastructure was worse than the team had understood. Monitoring that runs inside the cluster was not robust enough: when the cluster's DNS broke, it could not page anyone, and recovery took longer than it should have.

    View case
Check your cluster for this

These 16 cases involve Application database and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Faulty deployment1 case

Component
In-memory cacheApplication database
Contributing factor
Automation amplificationChange during peak hoursMonitoring blind spotNo runbookHealth check tied to a dependencyRetry storm
Organisation
Microsoft Azure
  1. IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hours
    Microsoft Azure2026

Capacity exhaustion10 cases

Component
Container memory limit3In-memory cache2CoreDNS2Application database2Kubernetes API server2Node pool2Cloud controller manager1Cluster autoscaler1Client-side DNS resolver and cache1etcd1GKE control plane1Horizontal pod autoscaler1kubelet1Cluster authentication provider1Persistent volumes and stateful sets1Pod scheduling and placement1VPC subnet or pod address range1AWS VPC CNI1
Contributing factor
Retry storm10Automation amplification3Unreviewed defaults3Monitoring blind spot3Single point of failure3Unknown limit2Limited debug access2Harmful manual action2No staged rollout2Ineffective rollback2Unverified assumption1Excessive blast radius1Ignored early signal1No alert on limit1No caching1No capacity planning1No runbook1No headroom1Unrepresentative staging1
Organisation
Scaleway2Buildkite1Fullstaq1Google Cloud1Honeycomb1Magic Pages1Neon1PostHog1Zalando1
  1. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch
    Buildkite2026
  2. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  3. IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletion
    Neon2025
  4. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
  5. IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PAR
    Scaleway2024
  6. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady
    Scaleway2024
  7. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  8. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022
  9. IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loop
    Fullstaq2019
  10. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019

Configuration change3 cases

Component
In-memory cache1Cloud load balancer1conntrack table1HAProxy1Ingress controller1Container memory limit1Persistent volumes and stateful sets1Pod priority and preemption1
Contributing factor
Retry storm3Unverified assumption2No headroom2Automation amplification1Excessive blast radius1Change with a delayed effect1No health probes1Partial rollout1Missed in review1Single point of failure1
Organisation
AdGuard VPN1Grafana Labs1LINE+1
  1. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
  2. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx
    LINE+2022
  3. IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus down
    Grafana Labs2019

Software defect2 cases

Component
Application database2Cloud load balancer1Cluster autoscaler1Authoritative DNS servers and records1Model hub1
Contributing factor
Retry storm2Automation amplification1No runbook1No headroom1Overlapping failures1
Organisation
Amazon Web Services1Hugging Face1
  1. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1
    Amazon Web Services2025
  2. IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three times
    Hugging Face2024