Cases4 cases

1 to 4 of 4

1. IL-0167 A phased rollout of an LKE software release in Osaka stalled LKE-E cluster creation, version upgrades and control plane ACL changes

   Linode2026

   Symptoms

   From about 19:15 UTC on 12 March 2026, customers in the Osaka data center (JP-OSA) could not create Linode Kubernetes Engine for Enterprise (LKE-E) clusters. Cluster creation stalled indefinitely.

   Cluster or node provisioning stallUpgrade stall

   Trigger

   A phased rollout of the LKE software release reached the Osaka data center.

   Root cause

   Faulty deployment

   Component

   Managed Kubernetes control plane and provider API

   Contributing factor

   –

   Detection

   The problem began at about 19:15 UTC, and the first status post, about five and a half hours later at 00:45 UTC on 13 March, said the team was investigating an emerging issue with deploying new LKE Enterprise clusters in Osaka.

   Remediation

   • deployed a software fix to mitigate the impact

   Action items

   • Investigate the root cause further and take preventive actions

   Lessons

   The problem followed a phased rollout of an LKE software release in the Osaka data center. There, cluster creation and administrative tasks stalled for more than six hours, and the first status post came about five and a half hours after the start.

   View case
2. IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down

   UK Ministry of Justice2025

   Symptoms

   At 09:36 on 18 March 2025, during the upgrade of the live cluster to EKS 1.30, users reported that their pods were being terminated and were not restarting automatically, in both production and non-production environments. Automated infrastructure alerts fired as well, and production services had outages.

   Pod evictionPods PendingService outageUpgrade stall

   Trigger

   Upgrading the live EKS cluster from 1.29 to 1.30, during which the descheduler attempted to reschedule all pods on the cordoned nodes simultaneously.

   Root cause

   Faulty deployment

   Component

   DeschedulerNode poolCluster autoscaler

   Contributing factor

   No headroomNo alert on limitUnreviewed defaults

   Detection

   Users reported at 09:36 that pods were being terminated and not restarting, and automated infrastructure alerts fired.

   Remediation

   • Removed the cordon annotation so pods could be rescheduled
   • Deleted stuck 1.30 nodes to force their recreation
   • Added a new node group of 20 nodes running 1.30 by hand
   • Cordoned and drained all old 1.29 nodes onto the new 1.30 nodes

   Action items

   • Raise an AWS Production System Down support case (Case ID: 174229643100496)
   • Get guidance from AWS support on querying control-plane logs to find the root cause of similar issues
   • Document a runbook for emergency scaling of a new node group in case of unexpected capacity shortages
   • Alert on an imbalance between the autoscaling group and the Kubernetes nodes
   • Review the descheduler configuration to understand why it throws pods off cordoned nodes, and fix it where possible
   • Alert on spikes in the pending pod count

   Lessons

   The write-up lists review actions rather than lessons. They show the team had no documented way to add an emergency node group, no alert on a mismatch between the autoscaling group and the nodes Kubernetes sees, no alert on a spike in pending pods, and no clear understanding of how the descheduler treats cordoned nodes.

   View case
3. IL-0164 An incremental Linode Kubernetes Engine production update that was missing changes from the previous release made cluster upgrades and node additions fail silently

   Linode2022

   Symptoms

   From about 18:30 UTC on June 15, 2022, customers of the Linode Kubernetes Engine (LKE) who tried to upgrade a cluster or add nodes saw those operations fail silently, without an error. The status page opened an investigation of an LKE issue at 19:54 UTC, and the affected list covered LKE in eleven regions.

   Upgrade stall

   Trigger

   An incremental update pushed to the LKE production environment at about 18:30 UTC.

   Root cause

   Faulty deployment

   Component

   Node pool

   Contributing factor

   –

   Detection

   The status page began investigating at 19:54 UTC, 1 hour 24 minutes after the update.

   Remediation

   • Released a follow-up LKE update at about 21:00 UTC that contained the missing changes

   Action items

   • Add steps and automation to the update process

   View case
4. IL-0149 Upgrades of AKS clusters on Virtual Machine Scale Sets failed with a 'platform image is not available' error after a service update left a backend service unhealthy

   Microsoft Azure2019

   Symptoms

   From 07:20 UTC on 28 September 2019, customers running Azure Kubernetes Service clusters on Virtual Machine Scale Sets (VMSS) got failures when they tried any cluster upgrade. The error read: "The platform image 'microsoft-aks:aks:aks-ubuntu-1604-201909:2019.08.21' is not available. Verify that all fields in the storage profile are correct." The issue title calls the error PlatformImageNotFound.

   Upgrade stall

   Trigger

   A recent service update affected instances of a backend service, which became unhealthy and stopped requests from completing.

   Root cause

   Faulty deployment

   Component

   Node operating system imageNode pool

   Contributing factor

   –

   Detection

   Customers received failure messages when they tried to upgrade their clusters.

   Remediation

   • published az vmss update commands that set the scale set's image version to 2019.09.25 and SKU to aks-ubuntu-1604-201909, after which the upgrade could be run again
   • mitigated affected clusters by hand (AKS and VMSS teams)
   • rolled out hot fixes to all regions, completed by 15 October 2019

   View case

By root cause1