Cases22 cases

15 to 19 of 22

1. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff

   Microsoft Azure2022

   Symptoms

   From 06:00 UTC on 30 August 2022, Azure VMs running Ubuntu 18.04 (bionic) with unattended-upgrades enabled began failing DNS resolution and losing network connectivity. Several Azure teams, among them the AKS and Azure Container Apps teams, were alerted by production alerts shortly after the update packages were published.

   DNS resolution failuresImage pull failuresMonitoring outage

   Trigger

   Unattended-upgrades on Ubuntu 18.04 VMs installed a Canonical security update published at 06:00 UTC on 30 August 2022, including systemd version 237-3ubuntu10.54.

   Root cause

   Breaking change in an upgrade

   Component

   Node operating system imagekubeletContainer registry and its credentialsClient-side DNS resolver and cache

   Contributing factor

   No staged rolloutExcessive blast radiusAutomation amplification

   Detection

   Several Azure teams, including AKS and Azure Container Apps, were alerted by production alerts shortly after the packages were published; incoming support cases then confirmed which Ubuntu versions were affected.

   Remediation

   • pointed customers to Canonical's description of the systemd bug and its fix
   • recommended rebooting affected VMs or running a template script that repairs DNS resolution
   • built automatic detection and remediation for AKS nodes and rolled it out across all regions
   • mitigated the remaining AKS customers manually through support

   Action items

   • Review the process that lets critical security updates bypass Safe Deployment Practices
   • Improve monitoring of the AKS data plane to alert on an uptick of errors
   • Review AKS monitoring algorithms so that similar VM errors are detected and alerted on
   • Establish an improved escalation path to Canonical during outages and add it to internal Technical Service Guides
   • AKS takes full control of the node security patch mechanism, with additional testing and controlled release of patches (estimated December 2022)
   • Work with Canonical to run dedicated tests on proposed packages before they are published for Azure IaaS VMs
   • Establish a process with Canonical to close the testing gap for the upgrade scenario (estimated March 2023)
   • Give AKS customers maintenance window control and environment staging for these patches (estimated December 2023)

   Lessons

   Security patches applied automatically by unattended-upgrades, once a day by default, and outside Azure's Safe Deployment Practices reached Ubuntu 18.04 VMs in every Azure region, so one systemd regression became a DNS outage across Azure. On AKS, lost node DNS showed up as image pull failures such as ImagePullBackoff, because kubelet could not resolve the registry. Azure plans to put node security patching under AKS's control with testing and controlled release; customers can use automatic node image upgrade for AKS, or automatic VM guest patching for VMs, instead of unattended-upgrades.

   View case
2. IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entries

   Preply2020

   Symptoms

   Prometheus detected a high rate of 5xx responses on three services, called A, B and C in the write-up, and paged the on-call engineers through VictorOps at 22:18 EET. The 5xx rate was also visible in Kibana.

   DNS resolution failuresHigh error rateStale endpoint routing

   Trigger

   Because load in the cluster was low, the CoreDNS autoscaler reduced the number of CoreDNS pods from 3 to 2.

   Root cause

   DNS or network fault

   Component

   CoreDNSkube-proxyconntrack table

   Contributing factor

   Automation amplificationIgnored early signalNo caching

   Detection

   Prometheus alerted on a high 5xx rate on services A, B and C and paged on-call through VictorOps.

   Remediation

   • reverted the latest release of the affected service
   • a regular deploy created new nodes, the CoreDNS autoscaler added CoreDNS pods, and the conntrack table was rewritten

   Action items

   • Disable the autoscaler for CoreDNS
   • Run a DNS cache server in production
   • Monitor conntrack usage

   Lessons

   Scaling CoreDNS down is not free: if kube-proxy fails to clear conntrack entries for the removed pod, clients keep sending DNS queries to it. Known DNS weaknesses in a cluster should be prioritised before they cause an outage, and fixes that only treat consequences leave the underlying conntrack problem in place.

   View case
3. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds

   Airbnb2019

   Symptoms

   On May 10 the latency graphs of several services showed a few spikes of up to about one second, then latency climbed to five seconds and then to 20 seconds, and it stayed there for a long time. The same pattern appeared on many services, not just one.

   High latencyIntermittent timeoutsLoad imbalanceDNS resolution failures

   Trigger

   Several services sent a high volume of external DNS queries, all of which went through kube-dns.

   Root cause

   Capacity exhaustion

   Component

   kube-dnskube-proxyClient-side DNS resolver and cache

   Contributing factor

   Unreviewed defaults

   Detection

   Latency graphs of several services showed the rise; engineers then looked at load per kube-dns pod.

   Lessons

   By default a pod discovers DNS through the kube-dns cluster IP and uses the ClusterFirst policy, so even external lookups go through kube-dns. If a pod does not need cluster DNS, set its dnsPolicy to Default or None. Check the DNS policy early and often. iptables picks backends at random and is not a load balancer, so pods behind a cluster IP, kube-dns included, can carry very different loads.

   View case
4. IL-0047 Pod-to-pod requests delayed by 5, 10 or 15 seconds because of DNS lookups lost to a conntrack race

   MindTickle2019

   Symptoms

   Pod-to-pod calls were intermittently slow and the web application felt slow overall. A request that pod A sent at time x reached pod B at x + 5s, x + 10s or x + 15s, always a multiple of five seconds.

   Intermittent timeoutsHigh latencyDNS resolution failures

   Trigger

   The C library resolver sent the A and AAAA queries for each lookup on the same port; where DNAT or SNAT translation takes place, a race condition in conntrack sometimes left one of the queries unanswered, and the resolver waited 5 seconds before retrying.

   Root cause

   DNS or network fault

   Component

   conntrack tableClient-side DNS resolver and cacheLinux kernel

   Contributing factor

   Unreviewed defaults

   Detection

   The team noticed intermittent delays in pod-to-pod calls and slowness in the web application, and that the delays were always multiples of five seconds.

   Remediation

   • added the single-request-reopen option to dnsConfig in the deployment specs
   • set GRPC_DNS_RESOLVER=native in the Dockerfiles of gRPC applications so they used the libc resolver
   • checked with tcpdump port 53 that A and AAAA queries left from different ports

   Lessons

   Delays that are exact multiples of five seconds point to DNS lookups being retried after the resolver's 5-second timeout. A resolver option set in the pod spec only helps processes that use the libc resolver; runtimes with their own resolver, such as gRPC with c-ares, ignore it.

   View case
5. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API

   Civis Analytics2019

   Symptoms

   After Civis released its new script infrastructure on Kubernetes internally in September 2018, developers reported that kubectl commands were slow, and users intermittently could not start notebooks and services.

   Control plane lagControl plane unresponsiveHigh latencyHigh error rateDNS resolution failures

   Trigger

   Migrating clients' Python, R and container scripts from the in-house Bocce system to Kubernetes (the project called Croquet) pushed the cluster to about 290 EC2 instances under a large script load.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serverTelemetry agentetcd

   Contributing factor

   Unknown limitUnverified assumptionMonitoring blind spotNo capacity planning

   Detection

   Developer reports of slow kubectl commands and user reports of notebooks and services failing to start, then high CPU on the master nodes.

   Remediation

   • moved internal users back to the old Bocce infrastructure
   • resized the three master nodes from m4.large to m4.4xlarge
   • changed the Datadog setup so per-node agents no longer polled the API server for service metadata
   • raised max-requests-inflight for non-mutating requests from 400 to 1200
   • grew the etcd EBS volumes from 20 to 100 GB (100 to 300 IOPS)
   • switched client scripts from the ClusterFirst DNS policy to Default

   Action items

   • Upgrade etcd from 2.2.1 to 3.x once Kops supports a safe migration
   • Solve the in-cluster DNS timeouts so client scripts can talk to other workloads in the cluster

   Lessons

   Kubernetes can reach thousands of nodes only after tuning, and third-party DaemonSets are a common cause of control plane load: an agent on every node that polls or watches the API server multiplies with the cluster size. Datadog, kube2iam and the fluentd metadata filter all showed this. A small configuration change often fixes it, but finding it took load testing and the API server access logs.

   View case

By root cause6