Cases36 cases

24 to 28 of 36

1. IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)

   2i2c2022

   Symptoms

   A user of the LIS JupyterHub reported a "backoff after failed scale-up" error, and user servers would not start. An engineer opened incident #36, "LIS hub cannot scale", at 5:40 AM Pacific on 14 November 2022.

   Scale-up failurePods PendingService outage

   Trigger

   At 10 AM UK time on 14 November 2022, the B zone of Google Cloud's europe-west2 region (London) ran out of resources to allocate. The LIS cluster's nodes were restricted to that zone.

   Root cause

   Cloud provider outage

   Component

   Node poolCluster autoscaler

   Contributing factor

   Single point of failurePages not delivered

   Detection

   A user reported the "backoff after failed scale-up" error; an engineer opened the incident at 5:40 AM Pacific, about three hours and forty minutes after the zone ran out of resources.

   Remediation

   • edited the node pool in the GCP console to allow node creation in the A and C zones of europe-west2, which were not exhausted

   Action items

   • Either restrict the node pool back to the B zone after the Black Friday event, or move the cluster and the NFS Filestore to the A zone (infrastructure issue #1944)
   • Investigate an enterprise plan that allows a regional NFS, at higher cost (infrastructure issue #1945)
   • Make sure PagerDuty notifications always get through to engineers (team-compass issue #574)

   Lessons

   A node pool pinned to one zone for storage performance cannot scale when the cloud provider runs out of capacity in that zone. The autoscaler's "backoff after failed scale-up" message usually points to quotas, and raising quotas does nothing here; the ZONE_RESOURCE_POOL_EXHAUSTED error in the logs is what identifies a zone without capacity. The review found that one engineer wearing many hats (incident commander, comms and debugging), with pages not reaching the other responders, made the response harder; a team response would have given more support.

   View case
2. IL-0164 An incremental Linode Kubernetes Engine production update that was missing changes from the previous release made cluster upgrades and node additions fail silently

   Linode2022

   Symptoms

   From about 18:30 UTC on June 15, 2022, customers of the Linode Kubernetes Engine (LKE) who tried to upgrade a cluster or add nodes saw those operations fail silently, without an error. The status page opened an investigation of an LKE issue at 19:54 UTC, and the affected list covered LKE in eleven regions.

   Upgrade stall

   Trigger

   An incremental update pushed to the LKE production environment at about 18:30 UTC.

   Root cause

   Faulty deployment

   Component

   Node pool

   Contributing factor

   –

   Detection

   The status page began investigating at 19:54 UTC, 1 hour 24 minutes after the update.

   Remediation

   • Released a follow-up LKE update at about 21:00 UTC that contained the missing changes

   Action items

   • Add steps and automation to the update process

   View case
3. IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effect

   Webhook Relay2022

   Symptoms

   At 12:00 EST the health check system reported that the backend APIs, the frontend and the tunnels in the EU region were down. The team first took it for a transient error, such as an interrupted client request or a server evicted for memory or CPU pressure, but the main dashboard was offline as well.

   Service outageControl plane lagControl plane unresponsiveStale or inconsistent API readsCredentials unavailable

   Root cause

   Certificate or credential expiry

   Component

   GKE control planeKubernetes API serverNode poolRBAC and service accounts

   Contributing factor

   Limited debug accessSingle point of failure

   Detection

   The health check system sent a notification that the service was down; the team then found the main dashboard offline.

   Remediation

   • recreated the node pools by hand after the credential rotation failed to recreate them
   • duplicated the main cluster's services into a newly created GKE cluster
   • deleted the Kubernetes Service records in the old cluster so its load balancers could be attached to the new cluster
   • detached the StatefulSet from the old cluster and switched all writes to the main disk
   • exported the data from the temporary ClickHouse instance and inserted it into the main instance

   Action items

   • Change the deployment manifests so services can be switched between clusters quickly without a complicated persistent disk migration

   Lessons

   Recovery of existing infrastructure needs a time limit. Throughout the outage the old cluster looked about to recover, and the reluctance to give up on the salvage effort kept the team from starting over with a new cluster much sooner, which would have been the right call. On older GKE clusters, check the validity of the cluster CA certificate, for example by decoding masterAuth.clusterCaCertificate from gcloud container clusters describe and reading it with openssl x509. GKE's multiple availability zones made no difference in this failure.

   View case
4. IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1

   GitLab2021

   Symptoms

   At 08:05 UTC on 2021-06-21 the API service's Apdex (the share of requests that finish within a target time) and Rails request queuing began trending the wrong way. At 09:01 the team found the API deployment unhealthy: it needed more pods for the day's growing traffic, but no new nodes could be added to give it pod capacity. Users saw slow responses; the review found no sign that error rates rose.

   Scale-up failureHigh latencyPods PendingDeploys failing

   Trigger

   Google Cloud ran out of SSD persistent disk stock in us-east1, the region GitLab.com runs in, while the API service needed more nodes for rising user traffic.

   Root cause

   Cloud provider outage

   Component

   Cluster autoscalerNode pool

   Contributing factor

   Monitoring blind spotLimited debug accessOverlapping failures

   Detection

   There was no alert on resource availability. Apdex and Rails queuing began trending the wrong way at 08:05 UTC and an engineer declared the incident in Slack at 08:43; finding the cause took a lot of log digging: cluster autoscaler errors about node pools in backoff, then ZONE_RESOURCE_POOL_EXHAUSTED at the instance-group level, and finally Google Cloud support naming SSD as the exhausted resource.

   Remediation

   • added a node pool that uses pd-standard disks
   • scaled the new node pool by hand across all three zones to relieve the API
   • brought more pd-standard node pools online for the other blocked workloads
   • stopped production deployments and opened a Google Cloud support case about the cluster autoscaler not working with the new node pools
   • changed node pools and nodeSelectors so the remaining workloads ran on pd-standard nodes
   • removed the SSD-backed node pools after a day without problems

   Action items

   • Move Kubernetes services to pd-standard disks (delivery#1837)
   • Capture resource exhaustion errors as a metric

   Lessons

   A cloud region can run out of one specific resource, here SSD persistent disks, and the only error may be a generic ZONE_RESOURCE_POOL_EXHAUSTED that does not name it. GitLab could neither monitor nor reserve resource availability, so detection took 22 hours and the cause came from Google Cloud support. The workloads ran fine on pd-standard disks. A second problem found on the way, a cluster autoscaler that does not fall back to another matching node pool when one cannot scale, slowed the move to the new node pools.

   View case
5. IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub

   2i2c2020

   Symptoms

   At 08:52 PM IST a report came in from WER that many students could not access the JupyterHub and that it was frozen; pages were stuck and nothing was usable. A rise in activity was visible by 09:02 PM, but the usual fixes, such as opening an incognito window and restarting servers, did not help.

   Service outageSustained high node CPU

   Trigger

   WER students' pods used a good chunk of their memory limit (tests had shown they almost always use just under 1G of their 2G limit, against a 512M guarantee) on nodes shared with the hub's core pods.

   Root cause

   Capacity exhaustion

   Component

   Node poolContainer memory limit

   Contributing factor

   No resource requests or limitsUnverified assumptionExcessive blast radius

   Detection

   WER reported stuck pages and students unable to reach the hub in a GitHub issue (2i2c-org/docs#27).

   Remediation

   • removed memory overcommit for the WER students
   • raised resource guarantees for all core pods, which restarted them and moved some to a new node

   Action items

   • Put user pods in a separate pool so they do not create pressure on the core pods (infrastructure#89)
   • Set limits on the support infrastructure (Prometheus, Grafana, ingress) as well (infrastructure#90)
   • Document and think about overcommit ratios for memory usage (infrastructure#91)
   • Set up better Grafana dashboards to monitor resource usage (infrastructure#92)
   • Document how people can get kubectl access to the cluster so others can look into issues too (infrastructure#87)

   Lessons

   Core pods without resource guarantees get whatever is left on a node, so when user pods on the same nodes fill the CPU and memory, the hub that serves everyone freezes. The memory overcommit ratio has to come from measured use: WER students used just under 1G against a 512M guarantee, which a 4x overcommit could not hold.

   View case

By root cause9