Cases2 cases

1 to 2 of 2

1. IL-0309 Preset's RDS PostgreSQL 14.7 to 17.9 upgrade in US East 1 ran out of storage and took Superset down for six hours

   Preset2026

   Symptoms

   A planned major version upgrade of Preset's production RDS PostgreSQL instances in US East 1, from 14.7 to 17.9, ran far past its maintenance window, and all customers on the us2a deployment could not reach Superset. After more than five hours the upgrade failed with "Postgres cluster is in a state where pg_upgrade cannot be completed successfully."

   Service outageUpgrade stall

   Trigger

   A planned major version upgrade of all RDS instances in production US East 1 from PostgreSQL 14.7 to 17.9, started with a ModifyDBInstance API call at 04:12 EDT.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase backups and snapshots

   Contributing factor

   Unrepresentative stagingUnverified assumptionUnknown limitLimited debug access

   Detection

   The planned upgrade was being watched as it ran: the snapshot's I/O was observed at 05:28 EDT, the incident was posted on the status page at 06:00 EDT and an AWS support ticket was opened at 07:26 EDT.

   Remediation

   • AWS rolled the failed upgrade back automatically, and the instance restarted on PostgreSQL 14.15 at 10:13 EDT

   Action items

   • Rehearse the upgrade against a full production-scale dataset to check storage and timing estimates before any future major version upgrade

   Lessons

   Preset learned that RDS takes a full, non-incremental snapshot before a major version upgrade, so the duration of the daily incremental snapshots and of the upgrades in other regions said nothing about how long the pre-upgrade snapshot would take on this instance. The S3 snapshot throughput limit of about 30–45MB/s only becomes visible during such full snapshots, which are rare in normal operation.

   View case
2. IL-0484 Slurm 24.05 to 25.05 upgrade broke GPU configuration on MIT CSAIL's cluster, and a failed downgrade kept it down

   MIT CSAIL2025

   Symptoms

   After a planned full day of downtime on Monday, September 8, 2025, to upgrade the MIT CSAIL Slurm compute cluster from Slurm 24.05 to Slurm 25.05, the Infrastructure Group found on Tuesday morning that the upgraded Slurm packages had been built without a feature the cluster needs for correct GPU configuration. The cluster did not return to normal operation until Wednesday morning, after a downgrade failed and the cluster was rebuilt.

   Upgrade stallService outage

   Trigger

   The upgrade from Slurm 24.05 to Slurm 25.05 on September 8 installed packages that were not compiled with a feature required for correct GPU configuration.

   Root cause

   Breaking change in an upgrade

   Component

   Slurm job schedulerDatabase backups and snapshots

   Contributing factor

   Ineffective rollbackUntested backup

   Detection

   On the Tuesday morning after the maintenance day, the team discovered that the upgraded Slurm packages were not compiled with the feature needed for GPU configuration.

   Remediation

   • Downgraded Slurm to the previous known-working version
   • Reinstalled the OS on all compute nodes and handled the nodes left in an intermediate state one by one
   • Re-deployed the cluster from scratch on a clean database

   Action items

   • Improve the testing and deployment processes based on the lessons of the outage

   Lessons

   The downgrade exposed flaws in the deployment and testing process: the deployment service could not reinstall every compute node at once, and the upgrade plan did not allow a simple downgrade or a working restore of the controller and database from backup.

   View case

By root cause2