Cases2 cases

1 to 2 of 2

1. IL-0185 A disk failure made the first control plane node of a Kubespray cluster read-only, and recovery needed the Kubespray inventory reordered and a new control plane node joined

   Hans H. Kim2023

   Symptoms

   The kernel on a master node of the Kubernetes cluster logged disk I/O errors: "[HAWKEYE] kernel:: [22329228.540857] blk_update_request: I/O error, dev sda, sector 795030104 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0". The sda disk switched to read-only, and the machine needed a new disk and an operating system reinstall.

   Node NotReady

   Trigger

   A hardware failure of the disk on the first control plane node.

   Root cause

   Hardware failure

   Component

   KubesprayetcdNode root diskConfigMaps and Secrets

   Contributing factor

   Single point of failure

   Detection

   Disk I/O errors from blk_update_request on dev sda appeared in the node's kernel log.

   Remediation

   • reordered the kube_control_plane group in the Kubespray inventory so that node-2 became the first control plane node
   • removed the failed machine from the cluster
   • edited the cluster-info ConfigMap in kube-public to point at a live control plane node and updated certificate-authority-data
   • installed and configured kubelet and the other components on a new node and joined it as a control plane node with kubeadm, skipping the etcd join phase
   • added the etcd member separately

   Lessons

   In a Kubespray cluster the first node of the kube_control_plane inventory group is special, so its failure needs its own procedure: move another node into the first position, fix the cluster-info ConfigMap, join a new control plane node and add the etcd member back. etcd should be brought back to full membership before another control plane node fails.

   View case
2. IL-0186 Pods on one Kubernetes worker node could not reach external HTTP and HTTPS because storage components had added nat PREROUTING rules that redirected ports 80 and 443 to local ports

   Hans H. Kim2023

   Symptoms

   On one worker node of a Kubernetes cluster, traffic to ports 80 and 443 went missing: pods on that node could not make outbound HTTP or HTTPS requests. Infrastructure engineers and the author's teammates debugged it.

   Misrouted requests

   Trigger

   A server on which components for storage work had been installed automatically added nat REDIRECT rules for ports 80 and 443 to the node's iptables.

   Root cause

   Configuration change

   Component

   Node iptables rules

   Contributing factor

   –

   Remediation

   • had the people responsible for the storage components delete the REDIRECT rules from the node

   Lessons

   Inside Kubernetes, kube-proxy itself manages iptables rules, so the author felt that part needs to be understood in depth. When packets go missing during network tests, check each step of the path in turn, which requires knowing clearly which steps the networking goes through.

   View case

By root cause2