Incident Library
Applied filters Root causeCapacity exhaustionComponentDatabase backups and snapshotsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0309 Preset's RDS PostgreSQL 14.7 to 17.9 upgrade in US East 1 ran out of storage and took Superset down for six hours
    Preset2026
    Symptoms

    A planned major version upgrade of Preset's production RDS PostgreSQL instances in US East 1, from 14.7 to 17.9, ran far past its maintenance window, and all customers on the us2a deployment could not reach Superset. After more than five hours the upgrade failed with "Postgres cluster is in a state where pg_upgrade cannot be completed successfully."

    Service outageUpgrade stall
    Trigger

    A planned major version upgrade of all RDS instances in production US East 1 from PostgreSQL 14.7 to 17.9, started with a ModifyDBInstance API call at 04:12 EDT.

    Root cause
    Capacity exhaustion
    Component
    DatabaseDatabase backups and snapshots
    Contributing factor
    Unrepresentative stagingUnverified assumptionUnknown limitLimited debug access
    Detection

    The planned upgrade was being watched as it ran: the snapshot's I/O was observed at 05:28 EDT, the incident was posted on the status page at 06:00 EDT and an AWS support ticket was opened at 07:26 EDT.

    Remediation
    • AWS rolled the failed upgrade back automatically, and the instance restarted on PostgreSQL 14.15 at 10:13 EDT
    Action items
    • Rehearse the upgrade against a full production-scale dataset to check storage and timing estimates before any future major version upgrade
    Lessons

    Preset learned that RDS takes a full, non-incremental snapshot before a major version upgrade, so the duration of the daily incremental snapshots and of the upgrades in other regions said nothing about how long the pre-upgrade snapshot would take on this instance. The S3 snapshot throughput limit of about 30–45MB/s only becomes visible during such full snapshots, which are rare in normal operation.

    View case
  2. IL-0453 Hakuna API latency spiked every day during the ElastiCache Redis backup as memory filled and the backup went forkless
    Hyperconnect2022
    Symptoms

    The service API latency of Hyperconnect's Hakuna rose sharply at a certain time, periodically. The per-dependency latency analysis in the APM showed that a bottleneck in Redis caused it, but no deployment and no change in traffic pattern could have affected how Redis was used.

    High latencyMemory pressure
    Trigger

    The daily ElastiCache backup on the primary node ran while available memory was insufficient, so ElastiCache used a forkless backup that slowed user requests.

    Root cause
    Capacity exhaustion
    Component
    In-memory cacheDatabase backups and snapshots
    Contributing factor
    No alert on limit
    Detection

    A latency graph showed the service API latency spiking periodically, and the APM's per-dependency latency analysis pointed at Redis.

    Remediation
    • Moved the ElastiCache backup from the primary node to the secondary node
    • Changed the code so data stored without a TTL is stored with one
    • Replaced the Redis Hash CrudRepository usages with a CustomRepository built on RedisTemplate
    • Ran a script that deleted the unneeded data already stored, freeing 58 GB on 3 databases
    Lessons

    The team says its Redis operation and monitoring had gaps, and lists what to watch: use a relational database when data has relationships or must be searched by value; write data that must be durable to on-disk storage rather than Redis; keep available memory at 50% or more when ElastiCache snapshots must be taken; take care that O(n) commands such as DEL on a collection do not stall production Redis; and check regularly for unnecessary I/O between the application and the database. Spring Data Redis repositories, which manage secondary indexes in the framework, suit workloads that need no TTL and need fast queries.

    View case
  3. IL-0334 Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went down
    Instapaper2017
    Symptoms

    On Wednesday, February 9, 2017 at 12:30PM PT, inserts of new entries into the bookmarks table of Instapaper's MySQL database on Amazon RDS started to fail with an error. The bookmarks table, which stores the articles Instapaper users save, had grown past 2TB, and the Instapaper service was down until Thursday, February 10 at 7:30pm PT.

    Writes rejectedService outage
    Trigger

    At 12:30PM PT on February 9 the bookmarks table grew past the 2TB file size limit of the instance's filesystem.

    Root cause
    Capacity exhaustion
    Component
    DatabaseDatabase backups and snapshotsDatabase replication
    Contributing factor
    Unknown limitMonitoring blind spotUntested backupNo runbookLimited debug access
    Remediation
    • Dumped the production database to a 5.5TB RAID-0 disk on an i2.8xlarge instance with Pinterest's SRE team
    • Launched a temporary production instance with limited access to archives
    • Set up an Aurora read replica of the old database as one recovery path
    • Imported the data into a new MySQL RDS instance without secondary indexes, then built the indexes
    • Had an Amazon engineer rsync the data from the ext3 filesystem to an ext4 filesystem mounted on the failed database
    • Synced the new ext4 database with the temporary production database through row-based replication
    • Promoted the new database to master and pointed the application at it
    Action items
    • Define a workflow that escalates system-wide Instapaper outages immediately to Pinterest's SRE team
    • Test MySQL backups every month instead of every three months
    • Put a process in place with Pinterest's SRE team to recover from this type of failure
    Lessons

    A filesystem limit that Instapaper did not know about and could not see made both the production database and all its backups useless, and the only way out was to restore the data onto a new filesystem. Because the hosted database could only be reached through MySQL, filesystem-level fixes such as rsync needed Amazon engineers. Even with a perfect response, rebuilding the database would have taken at least 10 hours. The lack of a disaster recovery plan for this kind of failure, and the estimate of the dump time from row counts, made the downtime and recovery longer than necessary, and the response could have drawn better on Pinterest's and AWS's people. The new escalation workflow and monthly backup tests would not have prevented this failure but would speed up the response to a future one.

    View case
Check your cluster for this

These 3 cases involve Database backups and snapshots and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion3 cases

Component
Database backups and snapshots3Database2In-memory cache1Database replication1
Contributing factor
Unknown limit2Limited debug access2Unverified assumption1Untested backup1Monitoring blind spot1No alert on limit1No runbook1Unrepresentative staging1
Organisation
Hyperconnect1Instapaper1Preset1
  1. IL-0309 Preset's RDS PostgreSQL 14.7 to 17.9 upgrade in US East 1 ran out of storage and took Superset down for six hours
    Preset2026
  2. IL-0453 Hakuna API latency spiked every day during the ElastiCache Redis backup as memory filled and the backup went forkless
    Hyperconnect2022
  3. IL-0334 Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went down
    Instapaper2017