Incident Library
Applied filters ComponentSlurm job schedulerFactorIneffective rollbackOrganisationMIT CSAILClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0484 Slurm 24.05 to 25.05 upgrade broke GPU configuration on MIT CSAIL's cluster, and a failed downgrade kept it down
    MIT CSAIL2025
    Symptoms

    After a planned full day of downtime on Monday, September 8, 2025, to upgrade the MIT CSAIL Slurm compute cluster from Slurm 24.05 to Slurm 25.05, the Infrastructure Group found on Tuesday morning that the upgraded Slurm packages had been built without a feature the cluster needs for correct GPU configuration. The cluster did not return to normal operation until Wednesday morning, after a downgrade failed and the cluster was rebuilt.

    Upgrade stallService outage
    Trigger

    The upgrade from Slurm 24.05 to Slurm 25.05 on September 8 installed packages that were not compiled with a feature required for correct GPU configuration.

    Root cause
    Breaking change in an upgrade
    Component
    Slurm job schedulerDatabase backups and snapshots
    Contributing factor
    Ineffective rollbackUntested backup
    Detection

    On the Tuesday morning after the maintenance day, the team discovered that the upgraded Slurm packages were not compiled with the feature needed for GPU configuration.

    Remediation
    • Downgraded Slurm to the previous known-working version
    • Reinstalled the OS on all compute nodes and handled the nodes left in an intermediate state one by one
    • Re-deployed the cluster from scratch on a clean database
    Action items
    • Improve the testing and deployment processes based on the lessons of the outage
    Lessons

    The downgrade exposed flaws in the deployment and testing process: the deployment service could not reinstall every compute node at once, and the upgrade plan did not allow a simple downgrade or a working restore of the controller and database from backup.

    View case
Check your cluster for this

This case involves Database backups and snapshots and Slurm job scheduler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade1 case

Component
Database backups and snapshotsSlurm job scheduler
Contributing factor
Untested backupIneffective rollback
Organisation
MIT CSAIL
  1. IL-0484 Slurm 24.05 to 25.05 upgrade broke GPU configuration on MIT CSAIL's cluster, and a failed downgrade kept it down
    MIT CSAIL2025