IL-0020 UK Ministry of Justice · 2025-03-18kubernetesaws
On 18 March 2025 the Ministry of Justice Cloud Platform team was upgrading the live EKS cluster from 1.29 to 1.30. During the upgrade the descheduler tried to reschedule all pods on the cordoned nodes at once, while some new 1.30 nodes were stuck. User pods were terminated and did not come back, which caused outages on production services.
From 10:15 the team removed the cordon annotation so pods could be rescheduled and deleted some stuck 1.30 nodes to force their recreation. Pods began recovering at 10:40 and services were operational again; the write-up gives the time to repair as 1h 37m. At 11:13 the team raised an AWS Production System Down support case.
To stabilise workloads and ensure redundancy the team added a new node group of 20 nodes on 1.30 by hand at 11:34, cordoned all old 1.29 nodes at 11:50 and drained them from 12:10. All nodes ran 1.30 by 16:16. The incident stayed open for clean-up and was closed at 12:09 on 20 March; the write-up gives the time to resolve as 2d 1h 29m. Why the descheduler throws pods off cordoned nodes is left to a follow-up review of its configuration.
At 09:36 on 18 March 2025, during the upgrade of the live cluster to EKS 1.30, users reported that their pods were being terminated and were not restarting automatically, in both production and non-production environments. Automated infrastructure alerts fired as well, and production services had outages.
Nodes had been cordoned for the upgrade, and the descheduler tried to reschedule all pods on the cordoned nodes at the same time. Some of the new 1.30 nodes were stuck and had to be deleted so they would be recreated. The Cloud Platform team began investigating at 09:50 and declared an incident at 10:03.
Root cause Faulty deployment
Users reported at 09:36 that pods were being terminated and not restarting, and automated infrastructure alerts fired.
The write-up lists review actions rather than lessons. They show the team had no documented way to add an emergency node group, no alert on a mismatch between the autoscaling group and the nodes Kubernetes sees, no alert on a spike in pending pods, and no clear understanding of how the descheduler treats cordoned nodes.
IL-0166 A compute host update removed the credential parser used to decode StackScripts, so new Linodes and every new LKE and LKE-E node failed to boot and autoscaling and node recycling failed in all regionsLinode · 2026same root cause · shared Cluster autoscaler, Node pool
IL-0163 An API change that slipped through a gap in automated testing made LKE clusters resize to the wrong number of nodes and added billable worker nodes to customer accountsLinode · 2023same root cause · shared Node pool
IL-0164 An incremental Linode Kubernetes Engine production update that was missing changes from the previous release made cluster upgrades and node additions fail silentlyLinode · 2022same root cause · shared Node pool
IL-0149 Upgrades of AKS clusters on Virtual Machine Scale Sets failed with a 'platform image is not available' error after a service update left a backend service unhealthyMicrosoft Azure · 2019same root cause · shared Node pool
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.