Cases3 cases

1 to 3 of 3

1. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts

   DFDS2022

   Symptoms

   On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.

   Node NotReadyUpgrade stallIntermittent timeoutsIntermittent connection failures

   Trigger

   At 07:30 UTC on 2022-10-31 the daily node rollover replaced the existing nodes with nodes built from AMI v20221027, the latest AWS EKS node AMI, released on 2022-10-29 and later recalled because of a kernel issue.

   Root cause

   Software defect

   Component

   Node operating system imageLinux kernelkubeletAWS VPC CNINode launch templateTerraform

   Contributing factor

   No staged rolloutAutomation amplificationOverlapping failuresMonitoring blind spotNo health probes

   Detection

   The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.

   Remediation

   • cordoned, drained and deleted 15 stuck nodes one at a time to complete the vpc-cni upgrade
   • changed the Terraform infrastructure-modules so a specific AMI could be set
   • rolled the EC2 Auto Scaling group launch configuration back to the previous known good AMI through the pipeline

   Action items

   • Reach out to teams with suggestions about readiness and liveness probes
   • Create a dashboard for readiness and liveness probe coverage
   • Write a wiki article on readiness and liveness probes and their usefulness
   • Modify infrastructure-modules to allow specifying a particular AMI
   • Schedule a discussion on whether to take the latest AMI immediately and how to test new AMIs before they are deployed
   • Implement alerting on abnormal NodeNotReady events and on increased socket use
   • Create a learning path for Kubernetes within the team
   • Collect logs from nodes centrally
   • Create a Prometheus exporter that retrieves the latest available AMI, with a dashboard and alerts

   Lessons

   When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters. Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.

   View case
2. IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failures

   AirMap2018

   Symptoms

   At about 15:00 UTC on 19 July 2018, monitoring alerted on rising 5xx responses from several APIs, most notably pilot and tiles. Logins failed first, because the pilot service fetches the user's profile, and from there all services became unresponsive. Every affected API ran on the production Kubernetes cluster.

   High error rateNode NotReadyDNS resolution failuresContainer crash loopService outage

   Root cause

   Software defect

   Component

   kubelet

   Contributing factor

   Unpatched known bug

   Detection

   Monitoring alerted on an increase in 5xx responses from several APIs, most notably pilot and tiles.

   Remediation

   • restarted the kubelets on the stuck worker nodes, which did not last
   • restarted all worker nodes in a staggered order starting at about 18:00 UTC, which took about 1.5 hours

   Action items

   • Restart nodes on a rolling schedule to keep nodes and kubelets healthy until the fixes land upstream and reach the cloud provider, Microsoft Azure

   Lessons

   When nodes keep falling back to NotReady after kubelet restarts and pods fail on network and DNS errors, a known upstream bug may be the cause; matching the symptoms to the upstream issue tracker, with help from the community, found the cause and the fix candidates.

   View case
3. IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod eviction

   Oath2018

   Symptoms

   During a rolling update of a kubelet change in one of Oath's on-premises production clusters, nodes started going NotReady, and soon all nodes in the cluster were NotReady.

   Node NotReadyPod evictionPods Pending

   Trigger

   A change to kubelet was rolled out across the nodes of one cluster; on restart, kubelet cleans up transient cgroup files, and a Kubernetes bug in that step left kubelet stuck instead of starting.

   Root cause

   Software defect

   Component

   kubeletKubernetes controller manager

   Contributing factor

   No health probesExcessive blast radius

   Detection

   Nodes reported NotReady and the eviction graphs climbed while the kubelet rollout was in progress.

   Remediation

   • failed the affected data center over so users were routed to the next available data center

   Action items

   • Run a functional test of kubelet after each upgrade batch and certify it before the next 30% of nodes are updated

   Lessons

   Check kubelet after upgrading it before moving to the next batch; a simple curl against the kubelet API would have caught this. The node controller's eviction throttling and full disruption mode, which the speakers note is not well documented, kept a cluster-wide kubelet failure from evicting every pod.

   View case

By root cause1