Incident Library
Applied filters ComponentDatabase backups and snapshotsSymptomUpgrade stallClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0309 Preset's RDS PostgreSQL 14.7 to 17.9 upgrade in US East 1 ran out of storage and took Superset down for six hours
    Preset2026
    Symptoms

    A planned major version upgrade of Preset's production RDS PostgreSQL instances in US East 1, from 14.7 to 17.9, ran far past its maintenance window, and all customers on the us2a deployment could not reach Superset. After more than five hours the upgrade failed with "Postgres cluster is in a state where pg_upgrade cannot be completed successfully."

    Service outageUpgrade stall
    Trigger

    A planned major version upgrade of all RDS instances in production US East 1 from PostgreSQL 14.7 to 17.9, started with a ModifyDBInstance API call at 04:12 EDT.

    Root cause
    Capacity exhaustion
    Component
    DatabaseDatabase backups and snapshots
    Contributing factor
    Unrepresentative stagingUnverified assumptionUnknown limitLimited debug access
    Detection

    The planned upgrade was being watched as it ran: the snapshot's I/O was observed at 05:28 EDT, the incident was posted on the status page at 06:00 EDT and an AWS support ticket was opened at 07:26 EDT.

    Remediation
    • AWS rolled the failed upgrade back automatically, and the instance restarted on PostgreSQL 14.15 at 10:13 EDT
    Action items
    • Rehearse the upgrade against a full production-scale dataset to check storage and timing estimates before any future major version upgrade
    Lessons

    Preset learned that RDS takes a full, non-incremental snapshot before a major version upgrade, so the duration of the daily incremental snapshots and of the upgrades in other regions said nothing about how long the pre-upgrade snapshot would take on this instance. The S3 snapshot throughput limit of about 30–45MB/s only becomes visible during such full snapshots, which are rare in normal operation.

    View case
  2. IL-0484 Slurm 24.05 to 25.05 upgrade broke GPU configuration on MIT CSAIL's cluster, and a failed downgrade kept it down
    MIT CSAIL2025
    Symptoms

    After a planned full day of downtime on Monday, September 8, 2025, to upgrade the MIT CSAIL Slurm compute cluster from Slurm 24.05 to Slurm 25.05, the Infrastructure Group found on Tuesday morning that the upgraded Slurm packages had been built without a feature the cluster needs for correct GPU configuration. The cluster did not return to normal operation until Wednesday morning, after a downgrade failed and the cluster was rebuilt.

    Upgrade stallService outage
    Trigger

    The upgrade from Slurm 24.05 to Slurm 25.05 on September 8 installed packages that were not compiled with a feature required for correct GPU configuration.

    Root cause
    Breaking change in an upgrade
    Component
    Slurm job schedulerDatabase backups and snapshots
    Contributing factor
    Ineffective rollbackUntested backup
    Detection

    On the Tuesday morning after the maintenance day, the team discovered that the upgraded Slurm packages were not compiled with the feature needed for GPU configuration.

    Remediation
    • Downgraded Slurm to the previous known-working version
    • Reinstalled the OS on all compute nodes and handled the nodes left in an intermediate state one by one
    • Re-deployed the cluster from scratch on a clean database
    Action items
    • Improve the testing and deployment processes based on the lessons of the outage
    Lessons

    The downgrade exposed flaws in the deployment and testing process: the deployment service could not reinstall every compute node at once, and the upgrade plan did not allow a simple downgrade or a working restore of the controller and database from backup.

    View case
Check your cluster for this

These 2 cases involve Database backups and snapshots and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Capacity exhaustion1 case

Component
DatabaseDatabase backups and snapshots
Contributing factor
Unverified assumptionUnknown limitLimited debug accessUnrepresentative staging
Organisation
Preset
  1. IL-0309 Preset's RDS PostgreSQL 14.7 to 17.9 upgrade in US East 1 ran out of storage and took Superset down for six hours
    Preset2026

Breaking change in an upgrade1 case

Component
Database backups and snapshotsSlurm job scheduler
Contributing factor
Untested backupIneffective rollback
Organisation
MIT CSAIL
  1. IL-0484 Slurm 24.05 to 25.05 upgrade broke GPU configuration on MIT CSAIL's cluster, and a failed downgrade kept it down
    MIT CSAIL2025