Cases5 cases

1 to 5 of 5

1. IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes

   2i2c2025

   Symptoms

   On 12 May 2025 at 20:15 EEST a Freshdesk ticket reported that users' servers on the Earthscope hub were taking more than 15 minutes to start and that their kernels were being killed. It happened during a workshop and also affected other instructors using the hub; a second report said a short course of about 30 users was having major problems.

   Pod eviction

   Trigger

   A workshop with about 30 users on a profile option that guaranteed 8 GB of memory and set no memory limit on 32 GB nodes.

   Root cause

   Capacity exhaustion

   Component

   Container memory limitNode pool

   Contributing factor

   No resource requests or limits

   Detection

   A Freshdesk support ticket from the community at 20:15 EEST, brought to the team's attention in the #support-freshdesk Slack channel at 20:49.

   Remediation

   • ran a helm rollout and hub redeploy, which partly improved server starts
   • made the 'medium' profile available and raised its memory guarantee to reduce overcommitment
   • pre-warmed the cluster with 15 nodes for the 30-user workshop
   • set explicit memory limits, with the guarantee equal to the limit, on the community's small-node profile options (https://github.com/2i2c-org/infrastructure/pull/6047)

   Action items

   • Write an incident report
   • Hold an internal retrospective about the incident (https://github.com/2i2c-org/meta/issues/2194)

   Lessons

   A memory guarantee without a limit lets a user server grow into memory that other pods on the same node were counting on, and on a shared node that ends with users' pods evicting each other. Setting the guarantee equal to the limit removes the overcommitment. For a known workshop, pre-warming enough nodes ahead of time keeps server start times short.

   View case
2. IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down

   UK Ministry of Justice2025

   Symptoms

   At 09:36 on 18 March 2025, during the upgrade of the live cluster to EKS 1.30, users reported that their pods were being terminated and were not restarting automatically, in both production and non-production environments. Automated infrastructure alerts fired as well, and production services had outages.

   Pod evictionPods PendingService outageUpgrade stall

   Trigger

   Upgrading the live EKS cluster from 1.29 to 1.30, during which the descheduler attempted to reschedule all pods on the cordoned nodes simultaneously.

   Root cause

   Faulty deployment

   Component

   DeschedulerNode poolCluster autoscaler

   Contributing factor

   No headroomNo alert on limitUnreviewed defaults

   Detection

   Users reported at 09:36 that pods were being terminated and not restarting, and automated infrastructure alerts fired.

   Remediation

   • Removed the cordon annotation so pods could be rescheduled
   • Deleted stuck 1.30 nodes to force their recreation
   • Added a new node group of 20 nodes running 1.30 by hand
   • Cordoned and drained all old 1.29 nodes onto the new 1.30 nodes

   Action items

   • Raise an AWS Production System Down support case (Case ID: 174229643100496)
   • Get guidance from AWS support on querying control-plane logs to find the root cause of similar issues
   • Document a runbook for emergency scaling of a new node group in case of unexpected capacity shortages
   • Alert on an imbalance between the autoscaling group and the Kubernetes nodes
   • Review the descheduler configuration to understand why it throws pods off cordoned nodes, and fix it where possible
   • Alert on spikes in the pending pod count

   Lessons

   The write-up lists review actions rather than lessons. They show the team had no documented way to add an emergency node group, no alert on a mismatch between the autoscaling group and the nodes Kubernetes sees, no alert on a spike in pending pods, and no clear understanding of how the descheduler treats cordoned nodes.

   View case
3. IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours

   CoW Protocol2024

   Symptoms

   At 16:40 UTC on 19 April 2024 CoW Protocol's mainnet production environment stopped producing settlements; the outage officially began then.

   Pods PendingIP address exhaustionPod evictionService outage

   Trigger

   During a cluster capacity expansion, the Infrastructure as Code automation updated the AWS VPC CNI plugin, because the CNI version was not hardcoded in the IaC and the IaC library version had recently changed.

   Root cause

   Configuration change

   Component

   AWS VPC CNINode pool

   Contributing factor

   Unpinned versionOverlapping failures

   Detection

   The CNI version mismatch was identified as the cause at 17:03 UTC, 23 minutes after settlements stopped.

   Remediation

   • upgraded the Kubernetes cluster
   • set up new node groups with a new version of the networking plugin

   Lessons

   Versions in IaC scripts should be specified explicitly so an automation run cannot update a component such as the CNI plugin unintentionally. Changes in production need change management that is easy to revert.

   View case
4. IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on device

   UK Ministry of Justice2023

   Symptoms

   At 13:42 the RootVolUtilisation-Critical alert fired in the high-priority alerts channel. At 14:03 a user reported image pull errors with the message "no space left on device".

   Node disk fullImage pull failuresPod evictionPods stuck in ContainerCreatingIP address exhaustion

   Trigger

   Upgrading the EKS Terraform module to version 18, which changed the node launch template, and draining workloads onto the new nodes it created.

   Root cause

   Configuration change

   Component

   Node root diskNode launch templateNode poolAWS VPC CNICoreDNS

   Contributing factor

   Missed in reviewNo configuration testsNo alert on limit

   Detection

   The RootVolUtilisation-Critical alert fired in the high-priority alerts channel at 13:42, and a user reported "no space left on device" image pull errors at 14:03.

   Remediation

   • Uncordoned the old default node group and drained the new nodes to move workloads back
   • Identified the launch template change that cut the root volume to 20 GB and applied a fix on the manager and live clusters
   • Drained the old node group onto the fixed nodes

   Action items

   • Update the runbook to compare launch template changes during an EKS module upgrade
   • Create a test setup that pulls images of different sizes, like live
   • Update the RootVolUtilisation alert runbook to check the disk space configuration
   • Scale CoreDNS dynamically based on the number of nodes
   • Investigate whether IPv6 can solve the IP prefix starvation problem
   • Add drift testing to identify when a terraform plan shows a change to the launch template
   • Set up logging for the CNI and ipamd logs and alerts on errors related to IP prefix starvation

   Lessons

   The write-up lists review actions rather than lessons. They aim at two gaps: the launch template changes brought by a module upgrade were not compared or drift-tested before rollout, and there were no CNI or ipamd logs or alerts to show the IP prefix starvation the team suspected during the drain.

   View case
5. IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up

   2i2c2023

   Symptoms

   The hub pod of the prod JupyterHub on the leap cluster (leap.2i2c.org) restarted unexpectedly during a large node scale-up, and a user opened a support ticket. The ingress-nginx and proxy pods kept running.

   Pod evictionPods Pending

   Trigger

   A user's dask-gateway workload added about 100 dask-worker nodes within a minute, and Prometheus started scraping the metrics exporters on about 200 nodes.

   Root cause

   Capacity exhaustion

   Component

   PrometheusNode pool

   Contributing factor

   No headroomExcessive blast radiusMonitoring blind spot

   Detection

   A user support ticket (2i2c Freshdesk ticket 414) about the leap hub; the engineer then looked at Grafana and the cluster's nodes and events.

   Action items

   • Configure CPU and memory requests for the hub and proxy pods (#2127; at least 100m CPU was proposed)
   • Make the Grafana OOMKiller dashboard work (#2213)
   • Move LEAP cloud infrastructure to shared nodes (#2209)
   • Update resource requests and limits, mainly for ingress-nginx and Prometheus (#2324)
   • Decide whether Prometheus should scrape dask worker pods (#2366)

   Lessons

   Prometheus load grows with the number of nodes, so a burst of autoscaled nodes can push a nearly full core node over the edge and evict the pods that matter most. Core pods that request 10m of CPU get a tenth of the CPU share of busy neighbours that request 100m. Kubernetes events expire after an hour, so evidence of OOM kills needs a dashboard that keeps it.

   View case

By root cause3