Cases36 cases

33 to 36 of 36

1. IL-0039 New Kubernetes worker nodes could not join any cluster because a package URL hardcoded in kops nodeup broke

   FREE NOW2019

   Symptoms

   At 16:00 CEST a developer reported in the #ask-platform Slack channel that a deployment to the Prelive cluster was failing, and a second report followed at 16:06.

   Pods PendingScale-up failure

   Trigger

   Roughly two to three days earlier, the CentOS repository mirror was restructured and container-selinux-2.68-1.el7.noarch.rpm moved, which made the static download URL hardcoded in kops nodeup invalid.

   Root cause

   Dependency failure

   Component

   Node operating system imageNode pool

   Contributing factor

   Monitoring blind spot

   Detection

   Developers asked for help in the #ask-platform Slack channel because their deployments were failing.

   Remediation

   • downloaded the package by hand on the Prelive nodes so their bootstrap could finish
   • changed the cluster AMI from Amazon Linux 2 to Debian and applied it to all clusters

   Action items

   • Create an alert that shows when nodes cannot join the cluster, possibly from information the cluster autoscaler already has
   • Document the external dependencies of node startup and look for ways to reduce them
   • Change the AMI back to Amazon Linux 2 Minimal once kops fixes the issue
   • Document node troubleshooting better: what happens during bootstrap and where the logs are

   Lessons

   Applying cluster changes through kops and Jenkins worked well, and troubleshooting quickly showed that the failing deployments came from missing cluster capacity. What went wrong was that nobody noticed, over a long period, that some autoscaling groups held more worker instances than Kubernetes had nodes. The team was lucky that no large number of production spot instances was terminated during the problem and that it was noticed during office hours.

   View case
2. IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPA

   loveholidays2019

   Symptoms

   On the evening of a long-overdue upgrade of a loveholidays production GKE cluster from 1.10.X to 1.12.5-gke.10, planned to take 2 to 2.5 hours, the master upgrade and the first two 7-node NodePools went smoothly. After that the NodePool upgrades slowed to a crawl. Each node is cordoned, drained and upgraded in turn. Even a 14-node cluster at about 5 minutes per node took over an hour, and with about 100 more nodes to go, some of them taking over 15 minutes to drain, the sequential drains were too slow to finish at a reasonable hour.

   Upgrade stallPods PendingScale-up failureService outageAutoscaler misbehaviour

   Trigger

   A manual upgrade of the GKE masters and NodePools from 1.10.X to 1.12.5-gke.10, two versions at once, on 19 March 2019; GKE's release notes of 14 March had already said that version was no longer available for new clusters or master upgrades because masters showed elevated error rates, but it could still be selected.

   Root cause

   Cloud provider outage

   Component

   GKE control planeNode poolCluster autoscalerIngress controllerHorizontal pod autoscaler

   Contributing factor

   Unrepresentative stagingOverlapping failuresIgnored early signal

   Detection

   The slow drains, zombie nodes and stuck NodePool were seen by the engineer running the upgrade; the broken Ingress was noticed after the upgrade and explained by Google support; the HPA problem was noticed the next day.

   Remediation

   • drained the remaining NodePool nodes ahead of the upgrade with kubectl drain --ignore-daemonsets --delete-local-data
   • deleted the zombie instances from the managed instance group with gcloud compute instance-groups managed delete-instances
   • cancelled the stuck UPGRADE_NODES operation with gcloud beta container operations cancel
   • used kubectl port-forward in place of the broken internal Ingresses
   • scaled down the older of two active ReplicaSets, or triggered a dummy deployment, to bring the HPA back

   Action items

   • Alert when a HorizontalPodAutoscaler is at 70% and at 100% of its capacity
   • Alert when a deployment has had two active ReplicaSets for 30 minutes and for 60 minutes
   • Re-evaluate termination grace periods that were set to arbitrary large values
   • Read the GKE release notes before every cluster upgrade
   • Use auto-upgrade in non-production environments, and manual cluster upgrades with automatic node upgrades in production
   • Keep environments as identical as possible, including resource names

   Lessons

   Read the release notes before upgrading: the target version had been withdrawn five days earlier. Drain time depends on pod configuration, so emptyDir volumes, long grace periods and missing disruption budgets turn a NodePool upgrade into hours. Pre-production only catches problems like the ingress bug if it matches production down to resource names. Auto-upgrades in non-production surface problems early, and a mix of manual cluster upgrades and automatic node upgrades in production keeps clusters current without much effort.

   View case
3. IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx pods

   DevOps Hof2019

   Symptoms

   While the old nodes of a GKE cluster were being drained to move workloads to a newly created node pool, none of the services in the cluster could be reached through ingress-nginx any more.

   Service outageProbe failures

   Trigger

   Cordoning and draining the old node pool removed those nodes from the GCP load balancer backend while the ingress-nginx pods were still running on them.

   Root cause

   Operator error

   Component

   Ingress controllerkube-proxyNode pool

   Contributing factor

   Unreviewed defaultsUnverified assumption

   Detection

   The team noticed during the drain that no service in the cluster was reachable through ingress-nginx.

   Remediation

   • deleted the nginx-ingress-controller pods still running on the old nodes so they were rescheduled on the new, ready nodes

   Lessons

   With externalTrafficPolicy: Local, the load balancer can only reach nodes that run an ingress pod, and cordoned nodes leave the load balancer backend immediately; if the ingress pods stay behind on cordoned nodes, all ingress traffic stops. Before any change that might touch traffic, look twice into the documentation.

   View case
4. IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a cluster

   Target2018

   Symptoms

   A Kubernetes cluster went down after one CronJob kept failing and its pods piled up. Because each failed run kept restarting instead of being replaced by the next one, the pods accumulated into the thousands. The speaker's arithmetic for a job scheduled every minute that fails for three days is 4,320 restarting pods.

   Service outage

   Trigger

   A CronJob kept failing.

   Root cause

   Configuration change

   Component

   Jobs and CronJobsNode pool

   Contributing factor

   Unreviewed defaultsAutomation amplification

   Lessons

   When deploying a CronJob, set its concurrency policy to Replace or Forbid; with the default, failing runs accumulate and restart together. An application should handle a missing dependency with probes and bounded back-off rather than crash in a loop, which also keeps it from hammering a dependency that is already struggling.

   View case

By root cause9