IL-0014 loveholidays · 2019-03-19kubernetesgcp
During the migration of loveholidays' on-premises applications to GCP the team had no time for routine cluster and NodePool upgrades and fell two versions behind. The same upgrade had been tested on several other clusters and environments without signs of trouble for production stability. A GKE upgrade runs in two steps, masters and then NodePools, and a NodePool upgrade drains and upgrades nodes one by one, so its length depends on the number of nodes. The cluster had many NodePools, some without active nodes.
The slow drains came from the cluster's own configuration. Pods with emptyDir volumes block drain because it cannot tell whether evicting them is safe; the cluster-autoscaler.kubernetes.io/safe-to-evict: "true" annotation allows it. Some applications had termination grace periods of 5 to 10 minutes, such as terminationGracePeriodSeconds: 600. A Pod Disruption Budget with maxUnavailable would let drain evict several pods at once. When it became clear the sequential drains would not finish at a reasonable hour, the engineer drained the remaining nodes ahead of time with kubectl drain --ignore-daemonsets --delete-local-data, which can evict more pods than a service tolerates; creating a new NodePool and moving pods to it is the alternative. The zombie nodes were cleared by mapping the managed instance group to its NodePool, listing its instances with gcloud compute instance-groups managed list-instances and deleting the stuck ones with delete-instances. The stuck upgrade was found with gcloud container operations list and describe and cancelled with gcloud beta container operations cancel, which leaves already upgraded nodes on the new version.
Google support explained the broken Ingress as a known bug that only hit Ingresses whose namespace plus name was between 44 and 47 characters long. The pre-production environments did not catch it because the production Ingress name was three characters longer; the fix shipped in GKE 1.12.6-gke.7 and the team used port-forward meanwhile. The cause of the HPA problem was not found. Scaling down the older of two active ReplicaSets, or triggering a dummy deployment by changing an environment variable, brought it back, and the team added Alertmanager alerts for HPAs near their maximum and for deployments with two active ReplicaSets. GKE had announced on 14 March that 1.12.5-gke.10 was withdrawn from new clusters and master upgrades because of elevated master error rates, and 1.12.6-gke.7, available from 26 March, patched the ingress bug and improved cluster autoscaling, so the author concludes several of these problems were preventable.
On the evening of a long-overdue upgrade of a loveholidays production GKE cluster from 1.10.X to 1.12.5-gke.10, planned to take 2 to 2.5 hours, the master upgrade and the first two 7-node NodePools went smoothly. After that the NodePool upgrades slowed to a crawl. Each node is cordoned, drained and upgraded in turn. Even a 14-node cluster at about 5 minutes per node took over an hour, and with about 100 more nodes to go, some of them taking over 15 minutes to drain, the sequential drains were too slow to finish at a reasonable hour.
One older NodePool turned out to be in a confused state: six nodes ran no pods but were never terminated. Pods stayed Pending, never landing on these zombie nodes and never triggering a scale-up. Another NodePool upgrade simply stopped with no sign of nodes draining. Its operation, of type UPGRADE_NODES with status RUNNING, reported "Done with 1 out of 20 nodes (5.0%): 1 being processed, 1 succeeded", and while it ran the console blocked any further change to the cluster, including cancelling the upgrade.
After the upgrade one of the internal convenience Ingresses stopped working, and its logs did not explain why. The next day the CPU-based HorizontalPodAutoscaler stopped reporting current CPU load and kept scaling up until it reached its maximum, which added load on downstream systems, cost money in underused nodes and slowed deployments. At times a single deployment had two active ReplicaSets for hours.
Symptom Upgrade stallPods Pending
Root cause Cloud provider outage
The slow drains, zombie nodes and stuck NodePool were seen by the engineer running the upgrade; the broken Ingress was noticed after the upgrade and explained by Google support; the HPA problem was noticed the next day.
Read the release notes before upgrading: the target version had been withdrawn five days earlier. Drain time depends on pod configuration, so emptyDir volumes, long grace periods and missing disruption budgets turn a NodePool upgrade into hours. Pre-production only catches problems like the ingress bug if it matches production down to resource names. Auto-upgrades in non-production surface problems early, and a mix of manual cluster upgrades and automatic node upgrades in production keeps clusters current without much effort.
IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)2i2c · 2022same root cause · shared Node pool, Cluster autoscaler
IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1GitLab · 2021same root cause · shared Node pool, Cluster autoscaler
IL-0020 linked by the editorhttps://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.