Cases1 case

1 to 1 of 1

1. IL-0039 New Kubernetes worker nodes could not join any cluster because a package URL hardcoded in kops nodeup broke

   FREE NOW2019

   Symptoms

   At 16:00 CEST a developer reported in the #ask-platform Slack channel that a deployment to the Prelive cluster was failing, and a second report followed at 16:06.

   Pods PendingScale-up failure

   Trigger

   Roughly two to three days earlier, the CentOS repository mirror was restructured and container-selinux-2.68-1.el7.noarch.rpm moved, which made the static download URL hardcoded in kops nodeup invalid.

   Root cause

   Dependency failure

   Component

   Node operating system imageNode pool

   Contributing factor

   Monitoring blind spot

   Detection

   Developers asked for help in the #ask-platform Slack channel because their deployments were failing.

   Remediation

   • downloaded the package by hand on the Prelive nodes so their bootstrap could finish
   • changed the cluster AMI from Amazon Linux 2 to Debian and applied it to all clusters

   Action items

   • Create an alert that shows when nodes cannot join the cluster, possibly from information the cluster autoscaler already has
   • Document the external dependencies of node startup and look for ways to reduce them
   • Change the AMI back to Amazon Linux 2 Minimal once kops fixes the issue
   • Document node troubleshooting better: what happens during bootstrap and where the logs are

   Lessons

   Applying cluster changes through kops and Jenkins worked well, and troubleshooting quickly showed that the failing deployments came from missing cluster capacity. What went wrong was that nobody noticed, over a long period, that some autoscaling groups held more worker instances than Kubernetes had nodes. The team was lucky that no large number of production spot instances was terminated during the problem and that it was noticed during office hours.

   View case

By root cause1