Cases5 cases

1 to 5 of 5

1. IL-0166 A compute host update removed the credential parser used to decode StackScripts, so new Linodes and every new LKE and LKE-E node failed to boot and autoscaling and node recycling failed in all regions

   Linode2026

   Symptoms

   Linodes deployed with StackScripts failed to deploy and could not boot. Every newly created Linode Kubernetes Engine (LKE) and LKE-E node relies on StackScripts, so LKE autoscaling and node recycling failed as well, in all regions.

   Scale-up failure

   Trigger

   A recent update on the compute hosts removed a credential parser that is used when decoding StackScripts.

   Root cause

   Faulty deployment

   Component

   Cluster autoscalerNode pool

   Contributing factor

   –

   Detection

   The issue was identified at 18:18 UTC on 2026-05-15. The first status post went up at 00:42 UTC on 2026-05-16.

   Remediation

   • tested and applied a rollback starting at 01:10 UTC on 2026-05-16, and failures dropped from 01:20 UTC

   Lessons

   Removing a credential parser from the compute hosts broke the boot of every Linode deployed with StackScripts, and every new LKE and LKE-E node is one of them, so node creation, autoscaling and node recycling stopped in all regions. The rollback reduced failures ten minutes after testing began.

   View case
2. IL-0167 A phased rollout of an LKE software release in Osaka stalled LKE-E cluster creation, version upgrades and control plane ACL changes

   Linode2026

   Symptoms

   From about 19:15 UTC on 12 March 2026, customers in the Osaka data center (JP-OSA) could not create Linode Kubernetes Engine for Enterprise (LKE-E) clusters. Cluster creation stalled indefinitely.

   Cluster or node provisioning stallUpgrade stall

   Trigger

   A phased rollout of the LKE software release reached the Osaka data center.

   Root cause

   Faulty deployment

   Component

   Managed Kubernetes control plane and provider API

   Contributing factor

   –

   Detection

   The problem began at about 19:15 UTC, and the first status post, about five and a half hours later at 00:45 UTC on 13 March, said the team was investigating an emerging issue with deploying new LKE Enterprise clusters in Osaka.

   Remediation

   • deployed a software fix to mitigate the impact

   Action items

   • Investigate the root cause further and take preventive actions

   Lessons

   The problem followed a phased rollout of an LKE software release in the Osaka data center. There, cluster creation and administrative tasks stalled for more than six hours, and the first status post came about five and a half hours after the start.

   View case
3. IL-0160 A repaired legacy GPU monitoring script sent a firmware query that triggers a race condition in the GPU System Processor of NVIDIA RTX 4000 Ada GPUs, locking up GPU Linodes and LKE worker nodes in several regions

   Linode2026

   Symptoms

   From about 21:00 UTC on March 4, 2026, customers using Linodes backed by NVIDIA RTX 4000 Ada GPUs saw the GPUs lock up. At first the problem seemed limited to worker nodes of the Linode Kubernetes Engine (LKE); later Linode confirmed that every Linode with this hardware was affected, while Standard Compute and other GPU instances were not.

   GPU unavailableControl plane unresponsive

   Trigger

   Deployment of a repaired legacy GPU monitoring script, fixed during a routine improvement effort to restore a missing metric on internal observability dashboards.

   Root cause

   Faulty deployment

   Component

   Telemetry agentGPU driver

   Contributing factor

   Missed in reviewUnverified assumption

   Detection

   Customers saw the lockups from about 21:00 UTC; the first status post went up at 02:07 UTC on March 5, about 5 hours after the lockups began.

   Remediation

   • Disabled the monitoring script across the GPU fleet
   • Rebooted the affected nodes

   Action items

   • Take improvement and prevention measures to stop a recurrence; the postmortem does not list them

   Lessons

   A monitoring script can take down the hardware it watches. A fix meant only to restore a missing metric brought back a firmware query written for an earlier GPU generation, and on the RTX 4000 Ada architecture that query against an active GPU triggers a lockup. The script ran across the GPU fleet, so every Linode with this GPU was affected in several regions, and the investigation first suspected a regression in the host hypervisor or the GPU firmware.

   View case
4. IL-0163 An API change that slipped through a gap in automated testing made LKE clusters resize to the wrong number of nodes and added billable worker nodes to customer accounts

   Linode2023

   Symptoms

   Resizing an LKE cluster produced an incorrect number of nodes, so some customer clusters were resized in unexpected ways and the extra worker nodes were billed to the customer accounts. The clusters themselves kept functioning. Linode's first status update, at 05:07 UTC on 1 June 2023, advised customers to avoid resizing their clusters if possible.

   Autoscaler misbehaviour

   Trigger

   At about 16:00 UTC on 31 May 2023 a change to the API that caused LKE clusters to scale improperly was pushed to production.

   Root cause

   Faulty deployment

   Component

   Node pool

   Contributing factor

   Gap in automated tests

   Detection

   The first status update, at 05:07 UTC on 1 June, said Linode had been made aware of an issue when resizing LKE clusters, about 13 hours after the change went out.

   Remediation

   • advised customers to avoid resizing their clusters
   • deployed a hotfix at 09:40 UTC on 1 June that corrected the scaling behaviour
   • left the extra worker nodes in place to avoid disrupting customer workloads
   • contacted affected customers by ticket about credit adjustments and further mitigation

   Action items

   • Audit the automated tests
   • Add pre-deployment checklist items so the same failure does not happen again

   Lessons

   An API change that alters scaling behaviour resized customer clusters and added cost while the clusters themselves kept working; Linode announced the issue about 13 hours after the change went out. Removing the extra nodes would have disrupted workloads, so the cost was handled through credits instead.

   View case
5. IL-0164 An incremental Linode Kubernetes Engine production update that was missing changes from the previous release made cluster upgrades and node additions fail silently

   Linode2022

   Symptoms

   From about 18:30 UTC on June 15, 2022, customers of the Linode Kubernetes Engine (LKE) who tried to upgrade a cluster or add nodes saw those operations fail silently, without an error. The status page opened an investigation of an LKE issue at 19:54 UTC, and the affected list covered LKE in eleven regions.

   Upgrade stall

   Trigger

   An incremental update pushed to the LKE production environment at about 18:30 UTC.

   Root cause

   Faulty deployment

   Component

   Node pool

   Contributing factor

   –

   Detection

   The status page began investigating at 19:54 UTC, 1 hour 24 minutes after the update.

   Remediation

   • Released a follow-up LKE update at about 21:00 UTC that contained the missing changes

   Action items

   • Add steps and automation to the update process

   View case

By root cause1