Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases29 cases

16 to 20 of 29

  1. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023
    Symptoms

    About two minutes after an engineer started the Kubernetes 1.23 to 1.24 upgrade of the cluster running Reddit's legacy stack, just after 19:00 UTC, the site stopped. The CDN edge request graph, which is kept separate from the cluster, showed traffic by status collapsing. Users saw the error page, a homepage with no posts, or plain errors.

    Service outageHigh error rateDNS resolution failuresMonitoring outagePods stuck in ContainerCreatingControl plane lag
    Trigger

    An in-place upgrade of the legacy cluster from Kubernetes 1.23 to 1.24; in 1.24 the node-role.kubernetes.io/master label is removed even from running clusters.

    Root cause
    Breaking change in an upgrade
    Component
    CalicoKubernetes API serverAdmission webhooketcd
    Contributing factor
    Configuration not in codeUnrepresentative stagingUntested backupMonitoring blind spotSingle point of failure
    Detection

    The site broke about two minutes into the upgrade; an incident was opened immediately and responders were on the call within three minutes, working from CDN edge graphs because the cluster's own metrics were gone.

    Remediation
    • deleted the Open Policy Agent webhook configurations to stop API server timeouts
    • terminated all worker nodes of the cluster
    • restored the control plane from the pre-upgrade backup onto the first control plane node
    • switched off the cluster autoscaler until the control plane was stable
    • generated new TLS certificates so replacement control plane nodes could join etcd
    • brought traffic back in steps from 1% to 100%
    • restarted legacy services that had been stopped by hand
    Action items
    • Codify cluster configuration so every choice has a record, and document in detail what cannot be codified
    • Standardise components across clusters and replace bespoke configurations
    • Build tooling to manage the lifecycle of whole clusters so they can be recreated or replicated
    • Add pre-production clusters that can be tested with confidence
    • Improve the backup restore process and make cluster upgrades safe
    Lessons

    Configuration that exists only in a live cluster, edited by hand through a special tool, can be forgotten, as the route reflector setup was, and then break on an upgrade nobody connects it to. A Kubernetes upgrade cannot be reverted without a restore, so the restore procedure has to be kept current and practised. Consistent, codified clusters are what make upgrades testable; bespoke clusters keep producing outages.

    View case
  2. IL-0230 kube-proxy sync failures during a Kubernetes upgrade corrupted node iptables, and CircleCI jobs stopped starting
    CircleCI2023
    Symptoms

    On 14 March 2023 at about 18:00 UTC, communication errors rose on one CircleCI service, and pipelines for all customers began to start late until jobs did not run at all. Within an hour the errors had spread to many services and to every function of the application, including the CircleCI UI and API.

    Service outageIntermittent connection failuresDNS resolution failures
    Trigger

    At about 17:40 UTC on 14 March the infrastructure team resumed the upgrade of the main production Kubernetes cluster with a changed approach for some components, after a first attempt on 13 March had been rolled back. The ongoing upgrade left kubelet and kube-proxy at mismatched versions.

    Root cause
    Breaking change in an upgrade
    Component
    kube-proxykubeletCoreDNSClient-side DNS resolver and cache
    Contributing factor
    Partial rolloutIneffective rollbackMonitoring blind spotHarmful manual action
    Detection

    At 18:00 UTC engineers saw communication errors rise on one service, then job processing delays climbing and job queues backing up across several services. The kube-proxy "Sync failed" errors were found only in the analysis after the incident.

    Remediation
    • Restarted every node in the cluster so each node built a fresh iptables ruleset
    • Restarted a separate pool of dependencies so jobs using contexts or the machine executor ran again
    • Added capacity to work through the backlog of jobs
    • Restarted the RabbitMQ nodes to fix a few corrupted queues
    Action items
    • Improve the Kubernetes upgrade process to limit the risk of future upgrades
    • Add metrics and alerts for kube-proxy sync errors
    • Evaluate ways to make the RabbitMQ infrastructure more resilient
    Lessons

    No single trigger caused the incident. Because networking underlies everything in the cluster, Kubernetes upgrades carry inherent risk, and CircleCI's risk mitigation did not hold up when several events interacted.

    View case
  3. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff
    Microsoft Azure2022
    Symptoms

    From 06:00 UTC on 30 August 2022, Azure VMs running Ubuntu 18.04 (bionic) with unattended-upgrades enabled began failing DNS resolution and losing network connectivity. Several Azure teams, among them the AKS and Azure Container Apps teams, were alerted by production alerts shortly after the update packages were published.

    DNS resolution failuresImage pull failuresMonitoring outage
    Trigger

    Unattended-upgrades on Ubuntu 18.04 VMs installed a Canonical security update published at 06:00 UTC on 30 August 2022, including systemd version 237-3ubuntu10.54.

    Root cause
    Breaking change in an upgrade
    Component
    Node operating system imagekubeletContainer registry and its credentialsClient-side DNS resolver and cache
    Contributing factor
    No staged rolloutExcessive blast radiusAutomation amplification
    Detection

    Several Azure teams, including AKS and Azure Container Apps, were alerted by production alerts shortly after the packages were published; incoming support cases then confirmed which Ubuntu versions were affected.

    Remediation
    • pointed customers to Canonical's description of the systemd bug and its fix
    • recommended rebooting affected VMs or running a template script that repairs DNS resolution
    • built automatic detection and remediation for AKS nodes and rolled it out across all regions
    • mitigated the remaining AKS customers manually through support
    Action items
    • Review the process that lets critical security updates bypass Safe Deployment Practices
    • Improve monitoring of the AKS data plane to alert on an uptick of errors
    • Review AKS monitoring algorithms so that similar VM errors are detected and alerted on
    • Establish an improved escalation path to Canonical during outages and add it to internal Technical Service Guides
    • AKS takes full control of the node security patch mechanism, with additional testing and controlled release of patches (estimated December 2022)
    • Work with Canonical to run dedicated tests on proposed packages before they are published for Azure IaaS VMs
    • Establish a process with Canonical to close the testing gap for the upgrade scenario (estimated March 2023)
    • Give AKS customers maintenance window control and environment staging for these patches (estimated December 2023)
    Lessons

    Security patches applied automatically by unattended-upgrades, once a day by default, and outside Azure's Safe Deployment Practices reached Ubuntu 18.04 VMs in every Azure region, so one systemd regression became a DNS outage across Azure. On AKS, lost node DNS showed up as image pull failures such as ImagePullBackoff, because kubelet could not resolve the registry. Azure plans to put node security patching under AKS's control with testing and controlled release; customers can use automatic node image upgrade for AKS, or automatic VM guest patching for VMs, instead of unattended-upgrades.

    View case
  4. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS
    Datadog2022
    Symptoms

    From September 2021, each rollout of an update to Datadog's metrics query service, which serves real-time and historical metric data to the web frontend and to monitor evaluation, came with a rise in errors. Clients retried failed queries, so few errors reached users, but the retries slowed dashboards and monitors. The service's logs showed DNS errors stopping it from connecting to the data stores it depends on.

    High error rateDNS resolution failuresHigh latencyOOMKilledPacket lossStale endpoint routing
    Trigger

    Rollouts of the metrics query service, which delete the old pods. The client behaviour behind the failures came from an August 2021 rollout of the metrics query clients that applied round_robin as the new default gRPC load balancing policy.

    Root cause
    Configuration change
    Component
    conntrack tableCiliumLinux kernelCoreDNSClient-side DNS resolver and cacheexternal-dnsHTTP and RPC client library
    Contributing factor
    Change with a delayed effectRetry stormUnknown limitMonitoring blind spot
    Detection

    Errors rose during rollouts of the metrics query service, and the service's logs showed DNS errors when connecting to its data stores.

    Remediation
    • Raised the memory of the node-local-dns pods from 64 MB to 256 MB, which stopped the OOM kills but not the DNS errors
    • Tested larger instance types and confirmed that scaling up the service's infrastructure resolved the errors
    • Removed the custom gRPC reconnect parameters from the clients to restore the defaults
    Action items
    • Contribute a change to Cilium that adds an unreachable route for a deleted pod's IP so clients get an ICMP error
    Lessons

    The errors looked like DNS but were not. ENA metrics and VPC Flow Logs gave the visibility into VPC connection tracking that host metrics did not, and Cilium, Kubernetes and AWS networking interact in ways that can make reverse path filtering drop packets it should allow. DNS propagation time depends on many settings. The causes were changes Datadog had made itself, to work around a dependency bug, to resolve earlier incidents and to override default reconnect parameters, which looked safe at the time and showed their effects much later, when the link was hard to see.

    View case
  5. IL-0517 Gitpod workspaces failed to start worldwide and lost data when CoreDNS lookups to 8.8.8.8 timed out in its clusters
    Gitpod2022
    Symptoms

    At around 5.44pm UTC on 25 January 2022, Gitpod saw a high error rate when pulling images from Google Container Registry (gcr.io), with the error "cannot get manifest". About 10 minutes later users reported that they could not start workspaces.

    Image pull failuresDNS resolution failuresService outageData loss
    Root cause
    DNS or network fault
    Component
    CoreDNSContainer registry and its credentialsObject storage
    Contributing factor
    Single point of failure
    Detection

    The team noticed a high error rate on image pulls from GCR at around 5.44pm UTC; about 10 minutes later users reported that workspaces would not start, and the on-call team started incident response.

    Remediation
    • Created new workspace clusters in the EU and the US with a werft job
    • Redirected traffic to the new clusters
    Action items
    • Keep workspace data on cloud provider persistent volumes that survive node and cluster failures
    • Let users download their files from a workspace that cannot be started
    • Improve the workspace backup process
    • Use three DNS servers instead of one: a GCP-internal one, 8.8.8.8 and Cloudflare's 1.1.1.1
    • Do more DNS resiliency work, tracked in an epic
    • Improve support for multi-region clusters and later go multi-cloud
    View case
Check your cluster for this

These 29 cases involve CoreDNS and Client-side DNS resolver and cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause6

Capacity exhaustion14 cases

Component
CoreDNS9Client-side DNS resolver and cache7Node pool4Container memory limit3etcd2Horizontal pod autoscaler2Kubernetes API server2kube-dns2kube-proxy2Linux kernel2VPC DNS resolver2CNI plugin1conntrack table1Container CPU limit1kubelet1Pod termination1Telemetry agent1
Contributing factor
Unreviewed defaults9Unknown limit5Monitoring blind spot3No caching3Unverified assumption2Retry storm2Single point of failure2Unrepresentative staging2Automation amplification1Excessive blast radius1Change during peak hours1Harmful manual action1No capacity planning1No resource requests or limits1No headroom1
Organisation
Airbnb1Backpackr1Buildkite1Chatwork1Civis Analytics1Cybozu1DigitalOcean1Mercari1MonotaRO1RebelMouse1Render1Tinder1unknown1Zalando1
  1. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch
    Buildkite2026
  2. IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodes
    DigitalOcean2026
  3. IL-0048 DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookups
    unknown2025
  4. IL-0080 kube-dns on Mercari's GKE clusters could not keep up with peak-hour DNS query rates, and a Sev1 production outage followed when several services failed DNS resolution
    Mercari2025
  5. IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse
    RebelMouse2025
  6. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down
    Backpackr2025
  7. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024
  8. IL-0440 Pods on Cybozu's Neco Kubernetes platform got frequent no such host and i/o timeout DNS errors from dropped packets
    Cybozu2024
  9. IL-0433 Shifting API traffic from EC2 to EKS on Fargate overloaded two CoreDNS pods and caused mass name-resolution errors
    MonotaRO2023
  10. IL-0445 Chatwork's service on EKS became unstable as conntrack tables on some nodes overflowed and connections were dropped
    Chatwork2021
  11. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds
    Airbnb2019
  12. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019
  13. IL-0472 Node-to-pod traffic and DNS failed on Tinder's 605-node Flannel cluster when every node's ARP cache passed gc_thresh3
    Tinder2019
  14. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019

Software defect4 cases

Component
Cloud load balancer1conntrack table1CoreDNS1Database1Client-side DNS resolver and cache1Authoritative DNS servers and records1Istio1kube-proxy1kubelet1
Contributing factor
Unreviewed defaults2Unverified assumption1Automation amplification1Unpatched known bug1No runbook1Retry storm1
Organisation
AirMap1Amazon Web Services1Cisco ThousandEyes1Toyota Connected1
  1. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1
    Amazon Web Services2025
  2. IL-0104 After a CoreDNS pod was OOMKilled, the talk names three causes: Istio's aggressive DNS cache refresh on one reused client port, kube-proxy not cleaning up conntrack entries of terminated pods, and the 2-minute UDP conntrack timeout
    Cisco ThousandEyes2025
  3. IL-0050 Pods on Kubernetes failed to resolve an external Redis hostname because of ndots:5, the musl resolver in Alpine images, and Netty's DNS resolver
    Toyota Connected2018
  4. IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failures
    AirMap2018

Breaking change in an upgrade4 cases

Component
Client-side DNS resolver and cache3kubelet2Admission webhook1Calico1ConfigMaps and Secrets1CoreDNS1etcd1GKE control plane1Kubernetes API server1kube-proxy1Node operating system image1Node pool1Persistent volumes and stateful sets1Container registry and its credentials1
Contributing factor
Monitoring blind spot2Ineffective rollback2Single point of failure2Automation amplification1Untested backup1Excessive blast radius1Configuration not in code1Harmful manual action1No headroom1No staged rollout1Partial rollout1Deferred upgrades1Unrepresentative staging1
Organisation
CircleCI1Microsoft Azure1Reddit1Wealthy1
  1. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025
  2. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023
  3. IL-0230 kube-proxy sync failures during a Kubernetes upgrade corrupted node iptables, and CircleCI jobs stopped starting
    CircleCI2023
  4. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff
    Microsoft Azure2022

Dependency failure1 case

Component
DatabaseAuthoritative DNS servers and records
Contributing factor
–
Organisation
Linode
  1. IL-0162 LKE users could not reach their cluster control planes or the LKE Dashboard because an earlier database update made the internal LKE DNS service hang
    Linode2025

Configuration change2 cases

Component
Client-side DNS resolver and cache2Cilium1conntrack table1CoreDNS1external-dns1HTTP and RPC client library1Kubernetes API server1Linux kernel1Telemetry agent1
Contributing factor
Monitoring blind spot2Change with a delayed effect1Cache-masked failure1Unknown limit1Limited debug access1No staged rollout1Retry storm1Unrepresentative staging1
Organisation
Datadog1OpenAI1
  1. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  2. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS
    Datadog2022

DNS or network fault4 cases

Component
CoreDNS3conntrack table2Client-side DNS resolver and cache2kube-proxy1Linux kernel1Object storage1Container registry and its credentials1Telemetry agent1
Contributing factor
Unreviewed defaults2Ignored early signal2Unverified assumption1Automation amplification1Unknown limit1No caching1Single point of failure1
Organisation
Adevinta1Gitpod1MindTickle1Preply1
  1. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS
    Adevinta2023
  2. IL-0517 Gitpod workspaces failed to start worldwide and lost data when CoreDNS lookups to 8.8.8.8 timed out in its clusters
    Gitpod2022
  3. IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entries
    Preply2020
  4. IL-0047 Pod-to-pod requests delayed by 5, 10 or 15 seconds because of DNS lookups lost to a conntrack race
    MindTickle2019