Cases12 cases

2 to 6 of 12

1. IL-0158 An expired credential caused authentication failures that stopped LKE and LKE-E from deploying clusters and nodes in all regions, which also blocked node recycling and cluster autoscaling

   Linode2026

   Symptoms

   From 11:45 UTC on 16 April 2026, customers of Linode Kubernetes Engine (LKE) and Linode Kubernetes Engine Enterprise (LKE-E) could not deploy new clusters or add new nodes to existing clusters. Because no new node could be deployed, node recycling and cluster autoscaling did not work either. The problem hit every region where LKE runs.

   Deploys failingScale-up failureCredentials unavailable

   Trigger

   A credential expired, which caused authentication failures when LKE and LKE-E deployed clusters and nodes. One status update describes it as an expired certificate.

   Root cause

   Certificate or credential expiry

   Component

   Node pool

   Contributing factor

   –

   Detection

   The first status page update, saying the team was investigating an emerging issue affecting LKE and LKE-E in all regions, was posted at 13:20 UTC.

   Remediation

   • created new credentials and deployed them, rotating the expired credential (certificate)

   Action items

   • Investigate what led to the credential expiry and take preventive actions

   Lessons

   One expired credential stopped every operation that needs a new node, in all regions at once: new clusters, added nodes, node recycling and autoscaling.

   View case
2. IL-0167 A phased rollout of an LKE software release in Osaka stalled LKE-E cluster creation, version upgrades and control plane ACL changes

   Linode2026

   Symptoms

   From about 19:15 UTC on 12 March 2026, customers in the Osaka data center (JP-OSA) could not create Linode Kubernetes Engine for Enterprise (LKE-E) clusters. Cluster creation stalled indefinitely.

   Cluster or node provisioning stallUpgrade stall

   Trigger

   A phased rollout of the LKE software release reached the Osaka data center.

   Root cause

   Faulty deployment

   Component

   Managed Kubernetes control plane and provider API

   Contributing factor

   –

   Detection

   The problem began at about 19:15 UTC, and the first status post, about five and a half hours later at 00:45 UTC on 13 March, said the team was investigating an emerging issue with deploying new LKE Enterprise clusters in Osaka.

   Remediation

   • deployed a software fix to mitigate the impact

   Action items

   • Investigate the root cause further and take preventive actions

   Lessons

   The problem followed a phased rollout of an LKE software release in the Osaka data center. There, cluster creation and administrative tasks stalled for more than six hours, and the first status post came about five and a half hours after the start.

   View case
3. IL-0160 A repaired legacy GPU monitoring script sent a firmware query that triggers a race condition in the GPU System Processor of NVIDIA RTX 4000 Ada GPUs, locking up GPU Linodes and LKE worker nodes in several regions

   Linode2026

   Symptoms

   From about 21:00 UTC on March 4, 2026, customers using Linodes backed by NVIDIA RTX 4000 Ada GPUs saw the GPUs lock up. At first the problem seemed limited to worker nodes of the Linode Kubernetes Engine (LKE); later Linode confirmed that every Linode with this hardware was affected, while Standard Compute and other GPU instances were not.

   GPU unavailableControl plane unresponsive

   Trigger

   Deployment of a repaired legacy GPU monitoring script, fixed during a routine improvement effort to restore a missing metric on internal observability dashboards.

   Root cause

   Faulty deployment

   Component

   Telemetry agentGPU driver

   Contributing factor

   Missed in reviewUnverified assumption

   Detection

   Customers saw the lockups from about 21:00 UTC; the first status post went up at 02:07 UTC on March 5, about 5 hours after the lockups began.

   Remediation

   • Disabled the monitoring script across the GPU fleet
   • Rebooted the affected nodes

   Action items

   • Take improvement and prevention measures to stop a recurrence; the postmortem does not list them

   Lessons

   A monitoring script can take down the hardware it watches. A fix meant only to restore a missing metric brought back a firmware query written for an earlier GPU generation, and on the RTX 4000 Ada architecture that query against an active GPU triggers a lockup. The script ran across the GPU fleet, so every Linode with this GPU was affected in several regions, and the investigation first suspected a regression in the host hypervisor or the GPU firmware.

   View case
4. IL-0156 Revoking old certificates after a rotation left a certificate key mismatch between API endpoints and the cluster controller, and LKE cluster management and autoscaling in Atlanta returned 5xx errors

   Linode2025

   Symptoms

   From about 02:29 UTC on 24 September 2025, customers of Linode Kubernetes Engine (LKE) in US-Southeast (Atlanta) could not manage their clusters, and autoscaling was impacted. Internal logs showed 5xx errors whenever customers tried to manage LKE clusters associated with the affected cluster controller.

   High error rateAutoscaler misbehaviour

   Trigger

   Old certificates on the LKE cluster controllers were revoked at 02:29 UTC on 24 September 2025, after the expiring certificates had been rotated on 23 September.

   Root cause

   Certificate or credential expiry

   Component

   Managed Kubernetes control plane and provider APICluster autoscaler

   Contributing factor

   Change with a delayed effect

   Remediation

   • deployed a fix for the certificate key mismatch
   • restarted the API endpoints

   Action items

   • Continue investigating the root cause and take preventive actions

   View case
5. IL-0162 LKE users could not reach their cluster control planes or the LKE Dashboard because an earlier database update made the internal LKE DNS service hang

   Linode2025

   Symptoms

   On 1 April 2025 users of Linode Kubernetes Engine (LKE) began having trouble connecting to their clusters' control planes, and the LKE Dashboard was affected as well. Services inside the clusters kept working. The first status update was posted at 20:28 UTC; at 21:33 UTC Linode said it was investigating an issue with DNS functionality within LKE.

   DNS resolution failuresControl plane unresponsive

   Trigger

   An earlier update to a database that the LKE DNS system depends on caused the DNS service to hang.

   Root cause

   Dependency failure

   Component

   Authoritative DNS servers and recordsApplication database

   Contributing factor

   –

   Detection

   The first status update, at 20:28 UTC on 1 April, said Linode was investigating and that users might have difficulty connecting to their cluster control plane. At 21:33 UTC Linode said the issue affected DNS functionality within LKE, and at 22:29 UTC that it had identified the underlying cause.

   Remediation

   • restarted the affected LKE DNS services

   Action items

   • Improve how the DNS services recover from transient connectivity failures to their dependencies
   • Improve monitoring to detect similar problems sooner
   • Update the deployment process to better coordinate changes between dependent systems

   Lessons

   A change to a database that the DNS service depends on can make the DNS service hang, which here cut off external access to the clusters and the LKE Dashboard while services inside the clusters kept working. The DNS service should recover by itself from transient failures of its dependencies, and changes to dependent systems need to be coordinated.

   View case

By root cause5