Cases2 cases

1 to 2 of 2

1. IL-0484 Slurm 24.05 to 25.05 upgrade broke GPU configuration on MIT CSAIL's cluster, and a failed downgrade kept it down

   MIT CSAIL2025

   Symptoms

   After a planned full day of downtime on Monday, September 8, 2025, to upgrade the MIT CSAIL Slurm compute cluster from Slurm 24.05 to Slurm 25.05, the Infrastructure Group found on Tuesday morning that the upgraded Slurm packages had been built without a feature the cluster needs for correct GPU configuration. The cluster did not return to normal operation until Wednesday morning, after a downgrade failed and the cluster was rebuilt.

   Upgrade stallService outage

   Trigger

   The upgrade from Slurm 24.05 to Slurm 25.05 on September 8 installed packages that were not compiled with a feature required for correct GPU configuration.

   Root cause

   Breaking change in an upgrade

   Component

   Slurm job schedulerDatabase backups and snapshots

   Contributing factor

   Ineffective rollbackUntested backup

   Detection

   On the Tuesday morning after the maintenance day, the team discovered that the upgraded Slurm packages were not compiled with the feature needed for GPU configuration.

   Remediation

   • Downgraded Slurm to the previous known-working version
   • Reinstalled the OS on all compute nodes and handled the nodes left in an intermediate state one by one
   • Re-deployed the cluster from scratch on a clean database

   Action items

   • Improve the testing and deployment processes based on the lessons of the outage

   Lessons

   The downgrade exposed flaws in the deployment and testing process: the deployment service could not reinstall every compute node at once, and the upgrade plan did not allow a simple downgrade or a working restore of the controller and database from backup.

   View case
2. IL-0449 Pods on Enigmo's new EKS managed node groups kept restarting or stuck Terminating and the nodes became unusable

   Enigmo2020

   Symptoms

   Two or three days after Enigmo created EKS managed node groups and moved its pods there from the old node groups, the cluster suddenly became unstable: pods restarted again and again or could not stop and stayed around indefinitely, node CPU and memory load rose high, and the nodes became unusable from the cluster.

   Pods stuck in TerminatingSustained high node CPUMemory pressureNode NotReadyUpgrade stall

   Trigger

   Enigmo created EKS managed node groups that used an AMI containing the bug and moved its pods onto them, as preparation for upgrading EKS from 1.15 to 1.16.

   Root cause

   Breaking change in an upgrade

   Component

   Node operating system imageNode pool

   Contributing factor

   Ineffective rollbackChange with a delayed effect

   Remediation

   • Changed the managed node groups' AMI version to the fixed release
   • Ran the managed node group update through CloudFormation
   • Watched the pods across all namespaces during the update
   • Force-deleted every pod that stayed in Terminating with kubectl delete --grace-period=0 --force

   Action items

   • Check the latest releases of the cluster's modules quarterly or monthly and upgrade them regularly

   Lessons

   As long as a team uses Amazon EKS or Kubernetes, it has to keep track of upgrades of the cluster and of modules such as ingress controllers, because a release with breaking changes can be announced without warning after a long stable period. Checking releases and upgrading on a regular schedule keeps the effect of each upgrade on the service small.

   View case

By root cause1