IL-0484 MIT CSAIL · 2025-09-08ai-infradatabase

Slurm 24.05 to 25.05 upgrade broke GPU configuration on MIT CSAIL's cluster, and a failed downgrade kept it down

On Monday, September 8, the Infrastructure Group took the CSAIL Slurm cluster down for a planned day of major software changes, including the multi-step upgrade from Slurm 24.05 to 25.05. Upgrading Slurm's database component took a long time, so the usual reboots and driver updates finished late in the evening. On Tuesday morning the team found that the upgraded packages lacked a feature needed for correct GPU configuration and chose to downgrade to the previous known-working version. Reaching a known state required an OS reinstall on all compute nodes, which exceeded the capacity of the deployment service and left most systems in an intermediate state that needed individual attention. Because of an oversight in the upgrade plan, downgrading the controller and database was not enough and the restore from backup did not succeed, so the team re-deployed the cluster from scratch on a clean database, and the scheduler was operating normally again by Wednesday morning.

Severity: –
Scope: Shared Slurm GPU compute cluster out of normal service from the planned maintenance on Monday until Wednesday morning
Users affected: –
Region: –
Time to detect: –
Time to resolve: –

Symptoms

After a planned full day of downtime on Monday, September 8, 2025, to upgrade the MIT CSAIL Slurm compute cluster from Slurm 24.05 to Slurm 25.05, the Infrastructure Group found on Tuesday morning that the upgraded Slurm packages had been built without a feature the cluster needs for correct GPU configuration. The cluster did not return to normal operation until Wednesday morning, after a downgrade failed and the cluster was rebuilt.

Symptom Upgrade stall Service outage

Root cause

Root cause Breaking change in an upgrade

Component Slurm job scheduler Database backups and snapshots

Contributing factor Ineffective rollback Untested backup

Trigger: The upgrade from Slurm 24.05 to Slurm 25.05 on September 8 installed packages that were not compiled with a feature required for correct GPU configuration.

Detection

On the Tuesday morning after the maintenance day, the team discovered that the upgraded Slurm packages were not compiled with the feature needed for GPU configuration.

Remediation

Downgraded Slurm to the previous known-working version
Reinstalled the OS on all compute nodes and handled the nodes left in an intermediate state one by one
Re-deployed the cluster from scratch on a clean database

Action items

Improve the testing and deployment processes based on the lessons of the outage

Lessons

The downgrade exposed flaws in the deployment and testing process: the deployment service could not reinstall every compute node at once, and the upgrade plan did not allow a simple downgrade or a working restore of the controller and database from backup.

Sources

https://tig.csail.mit.edu/shared-computing/slurm/post-mortem-sep2025 · blog

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Slurm job scheduler, Database backups and snapshots among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0484
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0484/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.