Cases2 cases

1 to 2 of 2

1. IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1

   GitLab2021

   Symptoms

   At 08:05 UTC on 2021-06-21 the API service's Apdex (the share of requests that finish within a target time) and Rails request queuing began trending the wrong way. At 09:01 the team found the API deployment unhealthy: it needed more pods for the day's growing traffic, but no new nodes could be added to give it pod capacity. Users saw slow responses; the review found no sign that error rates rose.

   Scale-up failureHigh latencyPods PendingDeploys failing

   Trigger

   Google Cloud ran out of SSD persistent disk stock in us-east1, the region GitLab.com runs in, while the API service needed more nodes for rising user traffic.

   Root cause

   Cloud provider outage

   Component

   Cluster autoscalerNode pool

   Contributing factor

   Monitoring blind spotLimited debug accessOverlapping failures

   Detection

   There was no alert on resource availability. Apdex and Rails queuing began trending the wrong way at 08:05 UTC and an engineer declared the incident in Slack at 08:43; finding the cause took a lot of log digging: cluster autoscaler errors about node pools in backoff, then ZONE_RESOURCE_POOL_EXHAUSTED at the instance-group level, and finally Google Cloud support naming SSD as the exhausted resource.

   Remediation

   • added a node pool that uses pd-standard disks
   • scaled the new node pool by hand across all three zones to relieve the API
   • brought more pd-standard node pools online for the other blocked workloads
   • stopped production deployments and opened a Google Cloud support case about the cluster autoscaler not working with the new node pools
   • changed node pools and nodeSelectors so the remaining workloads ran on pd-standard nodes
   • removed the SSD-backed node pools after a day without problems

   Action items

   • Move Kubernetes services to pd-standard disks (delivery#1837)
   • Capture resource exhaustion errors as a metric

   Lessons

   A cloud region can run out of one specific resource, here SSD persistent disks, and the only error may be a generic ZONE_RESOURCE_POOL_EXHAUSTED that does not name it. GitLab could neither monitor nor reserve resource availability, so detection took 22 hours and the cause came from Google Cloud support. The workloads ran fine on pd-standard disks. A second problem found on the way, a cluster autoscaler that does not fall back to another matching node pool when one cannot scale, slowed the move to the new node pools.

   View case
2. IL-0039 New Kubernetes worker nodes could not join any cluster because a package URL hardcoded in kops nodeup broke

   FREE NOW2019

   Symptoms

   At 16:00 CEST a developer reported in the #ask-platform Slack channel that a deployment to the Prelive cluster was failing, and a second report followed at 16:06.

   Pods PendingScale-up failure

   Trigger

   Roughly two to three days earlier, the CentOS repository mirror was restructured and container-selinux-2.68-1.el7.noarch.rpm moved, which made the static download URL hardcoded in kops nodeup invalid.

   Root cause

   Dependency failure

   Component

   Node operating system imageNode pool

   Contributing factor

   Monitoring blind spot

   Detection

   Developers asked for help in the #ask-platform Slack channel because their deployments were failing.

   Remediation

   • downloaded the package by hand on the Prelive nodes so their bootstrap could finish
   • changed the cluster AMI from Amazon Linux 2 to Debian and applied it to all clusters

   Action items

   • Create an alert that shows when nodes cannot join the cluster, possibly from information the cluster autoscaler already has
   • Document the external dependencies of node startup and look for ways to reduce them
   • Change the AMI back to Amazon Linux 2 Minimal once kops fixes the issue
   • Document node troubleshooting better: what happens during bootstrap and where the logs are

   Lessons

   Applying cluster changes through kops and Jenkins worked well, and troubleshooting quickly showed that the failing deployments came from missing cluster capacity. What went wrong was that nobody noticed, over a long period, that some autoscaling groups held more worker instances than Kubernetes had nodes. The team was lucky that no large number of production spot instances was terminated during the problem and that it was noticed during office hours.

   View case

By root cause2