Incident Library
Applied filters SymptomUpgrade stallFactorUnverified assumptionClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0309 Preset's RDS PostgreSQL 14.7 to 17.9 upgrade in US East 1 ran out of storage and took Superset down for six hours
    Preset2026
    Symptoms

    A planned major version upgrade of Preset's production RDS PostgreSQL instances in US East 1, from 14.7 to 17.9, ran far past its maintenance window, and all customers on the us2a deployment could not reach Superset. After more than five hours the upgrade failed with "Postgres cluster is in a state where pg_upgrade cannot be completed successfully."

    Service outageUpgrade stall
    Trigger

    A planned major version upgrade of all RDS instances in production US East 1 from PostgreSQL 14.7 to 17.9, started with a ModifyDBInstance API call at 04:12 EDT.

    Root cause
    Capacity exhaustion
    Component
    DatabaseDatabase backups and snapshots
    Contributing factor
    Unrepresentative stagingUnverified assumptionUnknown limitLimited debug access
    Detection

    The planned upgrade was being watched as it ran: the snapshot's I/O was observed at 05:28 EDT, the incident was posted on the status page at 06:00 EDT and an AWS support ticket was opened at 07:26 EDT.

    Remediation
    • AWS rolled the failed upgrade back automatically, and the instance restarted on PostgreSQL 14.15 at 10:13 EDT
    Action items
    • Rehearse the upgrade against a full production-scale dataset to check storage and timing estimates before any future major version upgrade
    Lessons

    Preset learned that RDS takes a full, non-incremental snapshot before a major version upgrade, so the duration of the daily incremental snapshots and of the upgrades in other regions said nothing about how long the pre-upgrade snapshot would take on this instance. The S3 snapshot throughput limit of about 30–45MB/s only becomes visible during such full snapshots, which are rare in normal operation.

    View case
Check your cluster for this

This case involves Database and Database backups and snapshots. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
DatabaseDatabase backups and snapshots
Contributing factor
Unverified assumptionUnknown limitLimited debug accessUnrepresentative staging
Organisation
Preset
  1. IL-0309 Preset's RDS PostgreSQL 14.7 to 17.9 upgrade in US East 1 ran out of storage and took Superset down for six hours
    Preset2026