Cases2 cases

1 to 2 of 2

1. IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down

   UK Ministry of Justice2025

   Symptoms

   At 09:36 on 18 March 2025, during the upgrade of the live cluster to EKS 1.30, users reported that their pods were being terminated and were not restarting automatically, in both production and non-production environments. Automated infrastructure alerts fired as well, and production services had outages.

   Pod evictionPods PendingService outageUpgrade stall

   Trigger

   Upgrading the live EKS cluster from 1.29 to 1.30, during which the descheduler attempted to reschedule all pods on the cordoned nodes simultaneously.

   Root cause

   Faulty deployment

   Component

   DeschedulerNode poolCluster autoscaler

   Contributing factor

   No headroomNo alert on limitUnreviewed defaults

   Detection

   Users reported at 09:36 that pods were being terminated and not restarting, and automated infrastructure alerts fired.

   Remediation

   • Removed the cordon annotation so pods could be rescheduled
   • Deleted stuck 1.30 nodes to force their recreation
   • Added a new node group of 20 nodes running 1.30 by hand
   • Cordoned and drained all old 1.29 nodes onto the new 1.30 nodes

   Action items

   • Raise an AWS Production System Down support case (Case ID: 174229643100496)
   • Get guidance from AWS support on querying control-plane logs to find the root cause of similar issues
   • Document a runbook for emergency scaling of a new node group in case of unexpected capacity shortages
   • Alert on an imbalance between the autoscaling group and the Kubernetes nodes
   • Review the descheduler configuration to understand why it throws pods off cordoned nodes, and fix it where possible
   • Alert on spikes in the pending pod count

   Lessons

   The write-up lists review actions rather than lessons. They show the team had no documented way to add an emergency node group, no alert on a mismatch between the autoscaling group and the nodes Kubernetes sees, no alert on a spike in pending pods, and no clear understanding of how the descheduler treats cordoned nodes.

   View case
2. IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 error

   UK Ministry of Justice2020

   Symptoms

   At 12:39 on 7 September 2020 the Cloud Platform team found that the Ingress API of the live-1 cluster was refusing every POST request. A user who provisioned a new service could not create an ingress into the cluster. The incident was declared at 12:54.

   Deploys failing

   Trigger

   Version 0.1.0 of the team's ingress controller module started creating a ValidatingWebhookConfiguration resource, and one new 0.1.0 ingress controller in the live-1 cluster failed to be created because of AWS resource limits.

   Root cause

   Faulty deployment

   Component

   Admission webhookIngress controller

   Contributing factor

   Single point of failure

   Detection

   The Ingress API was found refusing 100% of POST requests at 12:39; the incident was declared at 12:54.

   Remediation

   • removed by hand all the additional admission controllers created by module version 0.1.0
   • removed the admission webhook from the ingress controller module and released version 0.1.1
   • upgraded every ingress module on version 0.1.0 to 0.1.1

   Lessons

   Turning on the validating webhook in the shared ingress controller module made the webhook a single point of failure for every ingress controller pod in the ingress-controllers namespace: when one controller could not come up, no team could create an ingress.

   View case

By root cause1