Incident Library
Applied filters ComponentDatabase backups and snapshotsSymptomService outageFactorUnknown limitClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0309 Preset's RDS PostgreSQL 14.7 to 17.9 upgrade in US East 1 ran out of storage and took Superset down for six hours
    Preset2026
    Symptoms

    A planned major version upgrade of Preset's production RDS PostgreSQL instances in US East 1, from 14.7 to 17.9, ran far past its maintenance window, and all customers on the us2a deployment could not reach Superset. After more than five hours the upgrade failed with "Postgres cluster is in a state where pg_upgrade cannot be completed successfully."

    Service outageUpgrade stall
    Trigger

    A planned major version upgrade of all RDS instances in production US East 1 from PostgreSQL 14.7 to 17.9, started with a ModifyDBInstance API call at 04:12 EDT.

    Root cause
    Capacity exhaustion
    Component
    DatabaseDatabase backups and snapshots
    Contributing factor
    Unrepresentative stagingUnverified assumptionUnknown limitLimited debug access
    Detection

    The planned upgrade was being watched as it ran: the snapshot's I/O was observed at 05:28 EDT, the incident was posted on the status page at 06:00 EDT and an AWS support ticket was opened at 07:26 EDT.

    Remediation
    • AWS rolled the failed upgrade back automatically, and the instance restarted on PostgreSQL 14.15 at 10:13 EDT
    Action items
    • Rehearse the upgrade against a full production-scale dataset to check storage and timing estimates before any future major version upgrade
    Lessons

    Preset learned that RDS takes a full, non-incremental snapshot before a major version upgrade, so the duration of the daily incremental snapshots and of the upgrades in other regions said nothing about how long the pre-upgrade snapshot would take on this instance. The S3 snapshot throughput limit of about 30–45MB/s only becomes visible during such full snapshots, which are rare in normal operation.

    View case
  2. IL-0334 Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went down
    Instapaper2017
    Symptoms

    On Wednesday, February 9, 2017 at 12:30PM PT, inserts of new entries into the bookmarks table of Instapaper's MySQL database on Amazon RDS started to fail with an error. The bookmarks table, which stores the articles Instapaper users save, had grown past 2TB, and the Instapaper service was down until Thursday, February 10 at 7:30pm PT.

    Writes rejectedService outage
    Trigger

    At 12:30PM PT on February 9 the bookmarks table grew past the 2TB file size limit of the instance's filesystem.

    Root cause
    Capacity exhaustion
    Component
    DatabaseDatabase backups and snapshotsDatabase replication
    Contributing factor
    Unknown limitMonitoring blind spotUntested backupNo runbookLimited debug access
    Remediation
    • Dumped the production database to a 5.5TB RAID-0 disk on an i2.8xlarge instance with Pinterest's SRE team
    • Launched a temporary production instance with limited access to archives
    • Set up an Aurora read replica of the old database as one recovery path
    • Imported the data into a new MySQL RDS instance without secondary indexes, then built the indexes
    • Had an Amazon engineer rsync the data from the ext3 filesystem to an ext4 filesystem mounted on the failed database
    • Synced the new ext4 database with the temporary production database through row-based replication
    • Promoted the new database to master and pointed the application at it
    Action items
    • Define a workflow that escalates system-wide Instapaper outages immediately to Pinterest's SRE team
    • Test MySQL backups every month instead of every three months
    • Put a process in place with Pinterest's SRE team to recover from this type of failure
    Lessons

    A filesystem limit that Instapaper did not know about and could not see made both the production database and all its backups useless, and the only way out was to restore the data onto a new filesystem. Because the hosted database could only be reached through MySQL, filesystem-level fixes such as rsync needed Amazon engineers. Even with a perfect response, rebuilding the database would have taken at least 10 hours. The lack of a disaster recovery plan for this kind of failure, and the estimate of the dump time from row counts, made the downtime and recovery longer than necessary, and the response could have drawn better on Pinterest's and AWS's people. The new escalation workflow and monthly backup tests would not have prevented this failure but would speed up the response to a future one.

    View case
Check your cluster for this

These 2 cases involve Database and Database backups and snapshots. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion2 cases

Component
Database2Database backups and snapshots2Database replication1
Contributing factor
Unknown limit2Limited debug access2Unverified assumption1Untested backup1Monitoring blind spot1No runbook1Unrepresentative staging1
Organisation
Instapaper1Preset1
  1. IL-0309 Preset's RDS PostgreSQL 14.7 to 17.9 upgrade in US East 1 ran out of storage and took Superset down for six hours
    Preset2026
  2. IL-0334 Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went down
    Instapaper2017