IL-0163 Linode · 2023-05-31kubernetes
A change to the API made LKE clusters scale improperly. A gap in Linode's automated testing did not cover this failure case, so the change reached production at about 16:00 UTC on 31 May 2023. Affected customers' node pools were resized wrongly and accrued charges for the extra nodes.
Linode identified the cause at 08:56 UTC on 1 June and deployed a hotfix at 09:40 UTC. The extra worker nodes were left in place so that customer workloads would not be disrupted, and affected customers were contacted by ticket about credit adjustments and further mitigation. The incident was marked resolved at 21:23 UTC on 1 June.
Resizing an LKE cluster produced an incorrect number of nodes, so some customer clusters were resized in unexpected ways and the extra worker nodes were billed to the customer accounts. The clusters themselves kept functioning. Linode's first status update, at 05:07 UTC on 1 June 2023, advised customers to avoid resizing their clusters if possible.
Symptom Autoscaler misbehaviour
Root cause Faulty deployment
The first status update, at 05:07 UTC on 1 June, said Linode had been made aware of an issue when resizing LKE clusters, about 13 hours after the change went out.
An API change that alters scaling behaviour resized customer clusters and added cost while the clusters themselves kept working; Linode announced the issue about 13 hours after the change went out. Removing the extra nodes would have disrupted workloads, so the cost was handled through credits instead.
IL-0166 A compute host update removed the credential parser used to decode StackScripts, so new Linodes and every new LKE and LKE-E node failed to boot and autoscaling and node recycling failed in all regionsLinode · 2026same root cause · shared Node pool
IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went downUK Ministry of Justice · 2025same root cause · shared Node pool
IL-0164 An incremental Linode Kubernetes Engine production update that was missing changes from the previous release made cluster upgrades and node additions fail silentlyLinode · 2022same root cause · shared Node pool
IL-0149 Upgrades of AKS clusters on Virtual Machine Scale Sets failed with a 'platform image is not available' error after a service update left a backend service unhealthyMicrosoft Azure · 2019same root cause · shared Node pool
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.