Cases15 cases

6 to 10 of 15

1. IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down

   UK Ministry of Justice2025

   Symptoms

   At 09:36 on 18 March 2025, during the upgrade of the live cluster to EKS 1.30, users reported that their pods were being terminated and were not restarting automatically, in both production and non-production environments. Automated infrastructure alerts fired as well, and production services had outages.

   Pod evictionPods PendingService outageUpgrade stall

   Trigger

   Upgrading the live EKS cluster from 1.29 to 1.30, during which the descheduler attempted to reschedule all pods on the cordoned nodes simultaneously.

   Root cause

   Faulty deployment

   Component

   DeschedulerNode poolCluster autoscaler

   Contributing factor

   No headroomNo alert on limitUnreviewed defaults

   Detection

   Users reported at 09:36 that pods were being terminated and not restarting, and automated infrastructure alerts fired.

   Remediation

   • Removed the cordon annotation so pods could be rescheduled
   • Deleted stuck 1.30 nodes to force their recreation
   • Added a new node group of 20 nodes running 1.30 by hand
   • Cordoned and drained all old 1.29 nodes onto the new 1.30 nodes

   Action items

   • Raise an AWS Production System Down support case (Case ID: 174229643100496)
   • Get guidance from AWS support on querying control-plane logs to find the root cause of similar issues
   • Document a runbook for emergency scaling of a new node group in case of unexpected capacity shortages
   • Alert on an imbalance between the autoscaling group and the Kubernetes nodes
   • Review the descheduler configuration to understand why it throws pods off cordoned nodes, and fix it where possible
   • Alert on spikes in the pending pod count

   Lessons

   The write-up lists review actions rather than lessons. They show the team had no documented way to add an emergency node group, no alert on a mismatch between the autoscaling group and the nodes Kubernetes sees, no alert on a spike in pending pods, and no clear understanding of how the descheduler treats cordoned nodes.

   View case
2. IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours

   CoW Protocol2024

   Symptoms

   At 16:40 UTC on 19 April 2024 CoW Protocol's mainnet production environment stopped producing settlements; the outage officially began then.

   Pods PendingIP address exhaustionPod evictionService outage

   Trigger

   During a cluster capacity expansion, the Infrastructure as Code automation updated the AWS VPC CNI plugin, because the CNI version was not hardcoded in the IaC and the IaC library version had recently changed.

   Root cause

   Configuration change

   Component

   AWS VPC CNINode pool

   Contributing factor

   Unpinned versionOverlapping failures

   Detection

   The CNI version mismatch was identified as the cause at 17:03 UTC, 23 minutes after settlements stopped.

   Remediation

   • upgraded the Kubernetes cluster
   • set up new node groups with a new version of the networking plugin

   Lessons

   Versions in IaC scripts should be specified explicitly so an automation run cannot update a component such as the CNI plugin unintentionally. Changes in production need change management that is easy to revert.

   View case
3. IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together

   UK Ministry of Justice2023

   Symptoms

   At 10:23 on 6 June 2023 a user of the Cloud Platform reported that all of their production pods had been deleted at the same moment. Several users received Pingdom alerts that their applications were down for a few minutes.

   Service outage

   Trigger

   The team changed the instance type of the cluster's worker nodes through Terraform.

   Root cause

   Configuration change

   Component

   Node poolTerraform

   Contributing factor

   Excessive blast radiusNo runbook

   Detection

   Users reported that their production pods had been deleted at once and that Pingdom alerts showed their applications down; the incident was declared at 11:00.

   Action items

   • Ask AWS, through a support ticket, for the steps to change the node instance type without an outage
   • Plan how to make instance type changes through Terraform without downtime and add a runbook with the steps for changing the node instance type

   View case
4. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes

   DFDS2023sev1

   Symptoms

   At 09:33 UTC on 14 February 2023 end users told the platform team that Kubernetes logs were not arriving in CloudWatch as expected. The investigation went through fluentd and found that the nodes were running containerd as their container runtime instead of docker-shim. A scheduled node rollover the day before, at 07:30 UTC on 13 February, had set containerd as the default runtime by accident.

   Service outageResources deletedMonitoring outage

   Trigger

   A scheduled node rollover had accidentally made containerd the default container runtime, which stopped log shipping to CloudWatch; the unscheduled rollover to revert it timed out, and the pipeline run that followed reduced the Auto Scaling group from 30 to 15 instances.

   Root cause

   Configuration change

   Component

   Container runtimeNode launch templateNode poolTerraformTelemetry agent

   Contributing factor

   Automation amplificationUnreviewed defaultsUnrepresentative stagingNo configuration testsNo runbookHarmful manual action

   Detection

   End users reported the missing CloudWatch logs that started the work. The postmortem lists 2 minutes as the time to detection of the node failure.

   Remediation

   • ran the node rollover script to add new nodes, which failed on a timeout from Docker Hub
   • ran the node rollover script a second time, which added new nodes that the load balancer marked healthy at 11:01 UTC

   Action items

   • Document the rollover process, especially what can be done when it goes wrong
   • Investigate options for improving the rollover process
   • Add fluentd log collection to the QA test suite so logs are covered end to end
   • Add alarms for fluentd when too little or no log data is sent
   • Implement a staging environment
   • Implement guards against selecting an invalid container runtime

   Lessons

   When a change goes wrong, rolling back to a known working configuration is safer than rushing a fix, which can make the damage worse. Launching new worker nodes depends on the cloud-init script not failing in any way. The QA configuration differed from production, there was no staging environment, the QA tests did not cover fluentd log collection, and the rollover process had no documentation for when it fails.

   View case
5. IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)

   2i2c2022

   Symptoms

   A user of the LIS JupyterHub reported a "backoff after failed scale-up" error, and user servers would not start. An engineer opened incident #36, "LIS hub cannot scale", at 5:40 AM Pacific on 14 November 2022.

   Scale-up failurePods PendingService outage

   Trigger

   At 10 AM UK time on 14 November 2022, the B zone of Google Cloud's europe-west2 region (London) ran out of resources to allocate. The LIS cluster's nodes were restricted to that zone.

   Root cause

   Cloud provider outage

   Component

   Node poolCluster autoscaler

   Contributing factor

   Single point of failurePages not delivered

   Detection

   A user reported the "backoff after failed scale-up" error; an engineer opened the incident at 5:40 AM Pacific, about three hours and forty minutes after the zone ran out of resources.

   Remediation

   • edited the node pool in the GCP console to allow node creation in the A and C zones of europe-west2, which were not exhausted

   Action items

   • Either restrict the node pool back to the B zone after the Black Friday event, or move the cluster and the NFS Filestore to the A zone (infrastructure issue #1944)
   • Investigate an enterprise plan that allows a regional NFS, at higher cost (infrastructure issue #1945)
   • Make sure PagerDuty notifications always get through to engineers (team-compass issue #574)

   Lessons

   A node pool pinned to one zone for storage performance cannot scale when the cloud provider runs out of capacity in that zone. The autoscaler's "backoff after failed scale-up" message usually points to quotas, and raising quotas does nothing here; the ZONE_RESOURCE_POOL_EXHAUSTED error in the logs is what identifies a zone without capacity. The review found that one engineer wearing many hats (incident commander, comms and debugging), with pages not reaching the other responders, made the response harder; a team response would have given more support.

   View case

By root cause8