Incident Library

Organisation

MIT CSAIL

MIT CSAIL · mit-csail

1 case · 2025

Root cause Breaking change in an upgrade
Component Database backups and snapshotsSlurm job scheduler
Contributing factor Untested backupIneffective rollback
Symptom Service outageUpgrade stall

Cases

  1. IL-0484 Slurm 24.05 to 25.05 upgrade broke GPU configuration on MIT CSAIL's cluster, and a failed downgrade kept it downMIT CSAIL · 2025

    Root cause Breaking change in an upgrade