Incident Library
Applied filters ComponentDatabase replicationSymptomWrites rejectedFactorUnknown limitClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0334 Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went down
    Instapaper2017
    Symptoms

    On Wednesday, February 9, 2017 at 12:30PM PT, inserts of new entries into the bookmarks table of Instapaper's MySQL database on Amazon RDS started to fail with an error. The bookmarks table, which stores the articles Instapaper users save, had grown past 2TB, and the Instapaper service was down until Thursday, February 10 at 7:30pm PT.

    Writes rejectedService outage
    Trigger

    At 12:30PM PT on February 9 the bookmarks table grew past the 2TB file size limit of the instance's filesystem.

    Root cause
    Capacity exhaustion
    Component
    DatabaseDatabase backups and snapshotsDatabase replication
    Contributing factor
    Unknown limitMonitoring blind spotUntested backupNo runbookLimited debug access
    Remediation
    • Dumped the production database to a 5.5TB RAID-0 disk on an i2.8xlarge instance with Pinterest's SRE team
    • Launched a temporary production instance with limited access to archives
    • Set up an Aurora read replica of the old database as one recovery path
    • Imported the data into a new MySQL RDS instance without secondary indexes, then built the indexes
    • Had an Amazon engineer rsync the data from the ext3 filesystem to an ext4 filesystem mounted on the failed database
    • Synced the new ext4 database with the temporary production database through row-based replication
    • Promoted the new database to master and pointed the application at it
    Action items
    • Define a workflow that escalates system-wide Instapaper outages immediately to Pinterest's SRE team
    • Test MySQL backups every month instead of every three months
    • Put a process in place with Pinterest's SRE team to recover from this type of failure
    Lessons

    A filesystem limit that Instapaper did not know about and could not see made both the production database and all its backups useless, and the only way out was to restore the data onto a new filesystem. Because the hosted database could only be reached through MySQL, filesystem-level fixes such as rsync needed Amazon engineers. Even with a perfect response, rebuilding the database would have taken at least 10 hours. The lack of a disaster recovery plan for this kind of failure, and the estimate of the dump time from row counts, made the downtime and recovery longer than necessary, and the response could have drawn better on Pinterest's and AWS's people. The new escalation workflow and monthly backup tests would not have prevented this failure but would speed up the response to a future one.

    View case
Check your cluster for this

This case involves Database and Database backups and snapshots. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
DatabaseDatabase backups and snapshotsDatabase replication
Contributing factor
Untested backupUnknown limitLimited debug accessMonitoring blind spotNo runbook
Organisation
Instapaper
  1. IL-0334 Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went down
    Instapaper2017