Cases9 cases

2 to 6 of 9

1. IL-0158 An expired credential caused authentication failures that stopped LKE and LKE-E from deploying clusters and nodes in all regions, which also blocked node recycling and cluster autoscaling

   Linode2026

   Symptoms

   From 11:45 UTC on 16 April 2026, customers of Linode Kubernetes Engine (LKE) and Linode Kubernetes Engine Enterprise (LKE-E) could not deploy new clusters or add new nodes to existing clusters. Because no new node could be deployed, node recycling and cluster autoscaling did not work either. The problem hit every region where LKE runs.

   Deploys failingScale-up failureCredentials unavailable

   Trigger

   A credential expired, which caused authentication failures when LKE and LKE-E deployed clusters and nodes. One status update describes it as an expired certificate.

   Root cause

   Certificate or credential expiry

   Component

   Node pool

   Contributing factor

   –

   Detection

   The first status page update, saying the team was investigating an emerging issue affecting LKE and LKE-E in all regions, was posted at 13:20 UTC.

   Remediation

   • created new credentials and deployed them, rotating the expired credential (certificate)

   Action items

   • Investigate what led to the credential expiry and take preventive actions

   Lessons

   One expired credential stopped every operation that needs a new node, in all regions at once: new clusters, added nodes, node recycling and autoscaling.

   View case
2. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours

   Wealthy2025sev1

   Symptoms

   Nodes that already existed kept running, but every new node, spot or standard, on either v1.32 or v1.33, failed to initialise and could not join the cluster. Spot nodes that were preempted and recreated never came back, so services degraded progressively: background workers on spot instances, reporting services and order allocation stopped completely, and the APIs on existing nodes kept answering but could not scale for load spikes.

   Scale-up failureContainer crash loopService outageDNS resolution failures

   Trigger

   Google upgraded the GKE control plane from v1.32.x to v1.33.2 at the version deprecation deadline, although auto-upgrade was disabled on the node pools.

   Root cause

   Breaking change in an upgrade

   Component

   GKE control planeNode poolClient-side DNS resolver and cachePersistent volumes and stateful setsConfigMaps and Secrets

   Contributing factor

   Deferred upgradesSingle point of failureIneffective rollbackNo headroom

   Remediation

   • Created a new GKE cluster at v1.33, the version of the forced upgrade
   • Upgraded Istio to 1.27.1 for Kubernetes 1.33 compatibility
   • Updated the cert-manager configuration for Kubernetes 1.33
   • Migrated all production workloads to the new cluster and verified pod scheduling, DNS resolution, PVC mounting and external integrations
   • Decommissioned the broken cluster

   Action items

   • Configure all clusters to use the GKE stable release channel
   • Implement a separate management cluster for Argo CD, monitoring and other control components (Q4 2025)
   • Review spot instance usage for critical workloads and add on-demand fallback pools
   • Document the SRE upgrade procedure: management cluster first, production after a monitoring period of several days
   • Document a Kubernetes, Istio and cert-manager compatibility matrix
   • Create an automated upgrade testing pipeline
   • Reserve static IPs for quick migrations
   • Pin versions of Istio, cert-manager, CSI and DNS addons
   • Stay current with stable releases and regularly check GKE alerts and announcements to avoid deprecation deadlines

   Lessons

   GKE upgrades the control plane at a version's deprecation deadline even when auto-upgrade is disabled, so staying inside the supported version window is how to avoid a forced upgrade. A control plane upgrade can leave existing nodes healthy while every new node fails, and spot nodes that cannot rejoin after preemption turn this into cascading failures. Relying heavily on spot instances for critical workers turned a node join failure into a progressive outage. Kubernetes upgrades have to be planned together with dependent components such as Istio and cert-manager, and without a standby cluster on the same provider there was no immediate failover; the only recovery left was building a new cluster.

   View case
3. IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binaries

   Microsoft2025

   Symptoms

   Creating or scaling a node pool on AKS 1.31.8 failed: the VMSS instances never finished provisioning. This started after planned maintenance moved the node image from 202507.* to 202508.*, in a cluster where nodes must send all outbound traffic through an internal HTTP/HTTPS proxy and direct egress is blocked. It happened with both Ubuntu and AzureLinux images.

   Scale-up failureUpgrade stall

   Trigger

   A node image auto-upgrade from 202507.* to 202508.* during planned maintenance, on clusters whose Kubernetes patch version was not pre-cached in the new image.

   Root cause

   Software defect

   Component

   Node operating system imageNode pool

   Contributing factor

   Cache-masked failureAutomation amplification

   Detection

   The customer saw VMSS instance provisioning fail with a vmssCSE error in the VMSS activity logs and reported it as a GitHub issue.

   Remediation

   • The customer pinned the node image to a 202507.* snapshot on clusters running 1.31.8
   • The customer upgraded clusters to AKS 1.31.9 or 1.31.10, whose binaries are cached in the 202508.* images
   • The AKS team hot-patched VHD 20250811 so the proxy configuration is used for uncached binary downloads and made it globally available in public regions on 2025-09-04; it also announced the hotfix for VHD 20250820
   • The AKS team told users to reimage their nodes with az aks nodepool upgrade --node-image-only to pick up the hotfixed image

   Action items

   • Apply the proxy fix in all future VHDs

   Lessons

   A bootstrap path that runs only when an artefact is missing from a cache is easy to break without noticing: here only clusters on Kubernetes versions not pre-cached in the node image ever downloaded binaries, so only they met the proxy regression. Testing node provisioning behind a mandatory egress proxy, on older patch versions as well as the newest, would exercise that path. For clusters with restricted egress, pinning the node image or moving to a Kubernetes version cached in the image were both quick workarounds while the fix was shipped.

   View case
4. IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)

   2i2c2022

   Symptoms

   A user of the LIS JupyterHub reported a "backoff after failed scale-up" error, and user servers would not start. An engineer opened incident #36, "LIS hub cannot scale", at 5:40 AM Pacific on 14 November 2022.

   Scale-up failurePods PendingService outage

   Trigger

   At 10 AM UK time on 14 November 2022, the B zone of Google Cloud's europe-west2 region (London) ran out of resources to allocate. The LIS cluster's nodes were restricted to that zone.

   Root cause

   Cloud provider outage

   Component

   Node poolCluster autoscaler

   Contributing factor

   Single point of failurePages not delivered

   Detection

   A user reported the "backoff after failed scale-up" error; an engineer opened the incident at 5:40 AM Pacific, about three hours and forty minutes after the zone ran out of resources.

   Remediation

   • edited the node pool in the GCP console to allow node creation in the A and C zones of europe-west2, which were not exhausted

   Action items

   • Either restrict the node pool back to the B zone after the Black Friday event, or move the cluster and the NFS Filestore to the A zone (infrastructure issue #1944)
   • Investigate an enterprise plan that allows a regional NFS, at higher cost (infrastructure issue #1945)
   • Make sure PagerDuty notifications always get through to engineers (team-compass issue #574)

   Lessons

   A node pool pinned to one zone for storage performance cannot scale when the cloud provider runs out of capacity in that zone. The autoscaler's "backoff after failed scale-up" message usually points to quotas, and raising quotas does nothing here; the ZONE_RESOURCE_POOL_EXHAUSTED error in the logs is what identifies a zone without capacity. The review found that one engineer wearing many hats (incident commander, comms and debugging), with pages not reaching the other responders, made the response harder; a team response would have given more support.

   View case
5. IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1

   GitLab2021

   Symptoms

   At 08:05 UTC on 2021-06-21 the API service's Apdex (the share of requests that finish within a target time) and Rails request queuing began trending the wrong way. At 09:01 the team found the API deployment unhealthy: it needed more pods for the day's growing traffic, but no new nodes could be added to give it pod capacity. Users saw slow responses; the review found no sign that error rates rose.

   Scale-up failureHigh latencyPods PendingDeploys failing

   Trigger

   Google Cloud ran out of SSD persistent disk stock in us-east1, the region GitLab.com runs in, while the API service needed more nodes for rising user traffic.

   Root cause

   Cloud provider outage

   Component

   Cluster autoscalerNode pool

   Contributing factor

   Monitoring blind spotLimited debug accessOverlapping failures

   Detection

   There was no alert on resource availability. Apdex and Rails queuing began trending the wrong way at 08:05 UTC and an engineer declared the incident in Slack at 08:43; finding the cause took a lot of log digging: cluster autoscaler errors about node pools in backoff, then ZONE_RESOURCE_POOL_EXHAUSTED at the instance-group level, and finally Google Cloud support naming SSD as the exhausted resource.

   Remediation

   • added a node pool that uses pd-standard disks
   • scaled the new node pool by hand across all three zones to relieve the API
   • brought more pd-standard node pools online for the other blocked workloads
   • stopped production deployments and opened a Google Cloud support case about the cluster autoscaler not working with the new node pools
   • changed node pools and nodeSelectors so the remaining workloads ran on pd-standard nodes
   • removed the SSD-backed node pools after a day without problems

   Action items

   • Move Kubernetes services to pd-standard disks (delivery#1837)
   • Capture resource exhaustion errors as a metric

   Lessons

   A cloud region can run out of one specific resource, here SSD persistent disks, and the only error may be a generic ZONE_RESOURCE_POOL_EXHAUSTED that does not name it. GitLab could neither monitor nor reserve resource availability, so detection took 22 hours and the cause came from Google Cloud support. The workloads ran fine on pd-standard disks. A second problem found on the way, a cluster autoscaler that does not fall back to another matching node pool when one cannot scale, slowed the move to the new node pools.

   View case

By root cause7