Cases6 cases

5 to 6 of 6

1. IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutes

   GitLab2021

   Symptoms

   The web service breached its latency SLO at 07:30 UTC and its Apdex dropped: some requests to the GitLab.com web front end slowed down for about 15 minutes. This happened while web traffic was being shifted from virtual machines to the web pods on Kubernetes, and the pods did not scale up as the traffic arrived.

   High latencyAutoscaler misbehaviour

   Trigger

   Shifting web traffic from VMs to Kubernetes pods during the GitLab.com web migration.

   Root cause

   Configuration change

   Component

   Horizontal pod autoscaler

   Contributing factor

   Unverified assumption

   Detection

   The web latency SLO breach at 07:30 UTC; an engineer declared the incident in Slack at 07:40.

   Remediation

   • Lowered the target CPU utilisation of the web pods' autoscaler to a more conservative value (merge request gitlab-com/gl-infra/k8s-workloads/gitlab-com!1149)

   Action items

   • Adjust the target utilisation for the web pods (gitlab-com/gl-infra/k8s-workloads/gitlab-com!1149)

   Lessons

   When traffic is moved onto pods that rely on CPU-based autoscaling, a CPU target set too high can keep the pods from scaling as the traffic arrives; GitLab fixed it by setting a more conservative target.

   View case
2. IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPA

   loveholidays2019

   Symptoms

   On the evening of a long-overdue upgrade of a loveholidays production GKE cluster from 1.10.X to 1.12.5-gke.10, planned to take 2 to 2.5 hours, the master upgrade and the first two 7-node NodePools went smoothly. After that the NodePool upgrades slowed to a crawl. Each node is cordoned, drained and upgraded in turn. Even a 14-node cluster at about 5 minutes per node took over an hour, and with about 100 more nodes to go, some of them taking over 15 minutes to drain, the sequential drains were too slow to finish at a reasonable hour.

   Upgrade stallPods PendingScale-up failureService outageAutoscaler misbehaviour

   Trigger

   A manual upgrade of the GKE masters and NodePools from 1.10.X to 1.12.5-gke.10, two versions at once, on 19 March 2019; GKE's release notes of 14 March had already said that version was no longer available for new clusters or master upgrades because masters showed elevated error rates, but it could still be selected.

   Root cause

   Cloud provider outage

   Component

   GKE control planeNode poolCluster autoscalerIngress controllerHorizontal pod autoscaler

   Contributing factor

   Unrepresentative stagingOverlapping failuresIgnored early signal

   Detection

   The slow drains, zombie nodes and stuck NodePool were seen by the engineer running the upgrade; the broken Ingress was noticed after the upgrade and explained by Google support; the HPA problem was noticed the next day.

   Remediation

   • drained the remaining NodePool nodes ahead of the upgrade with kubectl drain --ignore-daemonsets --delete-local-data
   • deleted the zombie instances from the managed instance group with gcloud compute instance-groups managed delete-instances
   • cancelled the stuck UPGRADE_NODES operation with gcloud beta container operations cancel
   • used kubectl port-forward in place of the broken internal Ingresses
   • scaled down the older of two active ReplicaSets, or triggered a dummy deployment, to bring the HPA back

   Action items

   • Alert when a HorizontalPodAutoscaler is at 70% and at 100% of its capacity
   • Alert when a deployment has had two active ReplicaSets for 30 minutes and for 60 minutes
   • Re-evaluate termination grace periods that were set to arbitrary large values
   • Read the GKE release notes before every cluster upgrade
   • Use auto-upgrade in non-production environments, and manual cluster upgrades with automatic node upgrades in production
   • Keep environments as identical as possible, including resource names

   Lessons

   Read the release notes before upgrading: the target version had been withdrawn five days earlier. Drain time depends on pod configuration, so emptyDir volumes, long grace periods and missing disruption budgets turn a NodePool upgrade into hours. Pre-production only catches problems like the ingress bug if it matches production down to resource names. Auto-upgrades in non-production surface problems early, and a mix of manual cluster upgrades and automatic node upgrades in production keeps clusters current without much effort.

   View case

By root cause5