IL-0334 Instapaper · 2017-02-09databaseaws
Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went down
MySQL RDS instances created before April 2014 run on an ext3 filesystem with a 2TB file size limit; later instances use ext4 with a 6TB limit. Instapaper's first RDS instance was created in June 2013, when betaworks moved Instapaper from Softlayer to AWS with two devops contractors; any knowledge of the limit probably left with them. In March 2015, to fix performance problems during the backup window, the team created a read replica of that instance, upgraded it from MySQL v5.6.18 and cut over to it with about 5 minutes of downtime. Although created after April 2014, the replica inherited the original instance's filesystem and its 2TB limit. On February 9, 2017 the bookmarks table passed 2TB and inserts failed. Every backup was a filesystem snapshot under the same limit, and there was no disaster recovery plan for this kind of failure.
Guided by Pinterest's SRE team, Instapaper dumped the 2.5TB production database to a 5.5TB RAID-0 disk on an i2.8xlarge instance. When the dump proved far slower than estimated, the team switched to a contingency plan and, in about six hours, built a temporary production instance with limited access to archives, which went live after 31 hours of downtime. Over the weekend RDS engineers worked on two parallel paths: an Aurora read replica of the old database, finished in about 24 hours but risky because Aurora had not been tested with the Instapaper codebase, and a new MySQL RDS instance loaded without secondary indexes in 8 hours, where each of the three secondary indexes then took about 16 hours. Because the indexes were too slow, an Amazon engineer mounted an ext4 filesystem on the failed production database and copied the data across with rsync in about 8 hours, giving an ext4-backed instance with all data and indexes.
RDS engineers then set up row-based replication from the binary logs of the temporary production database to the new ext4 instance, which took about three hours to catch up on the changes made since Thursday. The new database was promoted to master and the application was deployed to point at it. Full service was restored on February 14 without losing any older articles, recent changes or articles saved after the outage.
- Severity
- –
- Scope
- Instapaper service down for 31 hours, then five days of limited access to archives until full recovery on February 14; no user articles were lost
- Users affected
- Instapaper users
- Region
- –
- Time to detect
- –
- Time to resolve
- 31h
Symptoms
On Wednesday, February 9, 2017 at 12:30PM PT, inserts of new entries into the bookmarks table of Instapaper's MySQL database on Amazon RDS started to fail with an error. The bookmarks table, which stores the articles Instapaper users save, had grown past 2TB, and the Instapaper service was down until Thursday, February 10 at 7:30pm PT.
Nothing on the RDS side warned of the problem. As far as Instapaper could tell, the RDS console offered no monitoring, alert or log that showed the table approaching a 2TB file size limit, or that the instance was subject to such a limit at all, and even after the outage nothing marked the database as having a critical issue. The ten days of automated daily RDS backups were filesystem snapshots with the same limit, so none of them could be used to recover.
After a long call with AWS support and talks with Pinterest's Site Reliability engineers, the only way forward was a full dump and restore of the 2.5TB database to a new instance. Instapaper had estimated six to eight hours for the dump from the number of rows, but the first 25 percent of bookmarks held only 10 percent of the data: the first dump took 24 hours and a second, parallelised one took 10 hours.
Symptom Writes rejectedService outage
Root cause
Root cause Capacity exhaustion
- Trigger
- At 12:30PM PT on February 9 the bookmarks table grew past the 2TB file size limit of the instance's filesystem.
Detection
–
Remediation
- Dumped the production database to a 5.5TB RAID-0 disk on an i2.8xlarge instance with Pinterest's SRE team
- Launched a temporary production instance with limited access to archives
- Set up an Aurora read replica of the old database as one recovery path
- Imported the data into a new MySQL RDS instance without secondary indexes, then built the indexes
- Had an Amazon engineer rsync the data from the ext3 filesystem to an ext4 filesystem mounted on the failed database
- Synced the new ext4 database with the temporary production database through row-based replication
- Promoted the new database to master and pointed the application at it
Action items
- Define a workflow that escalates system-wide Instapaper outages immediately to Pinterest's SRE team
- Test MySQL backups every month instead of every three months
- Put a process in place with Pinterest's SRE team to recover from this type of failure
Lessons
A filesystem limit that Instapaper did not know about and could not see made both the production database and all its backups useless, and the only way out was to restore the data onto a new filesystem. Because the hosted database could only be reached through MySQL, filesystem-level fixes such as rsync needed Amazon engineers. Even with a perfect response, rebuilding the database would have taken at least 10 hours. The lack of a disaster recovery plan for this kind of failure, and the estimate of the dump time from row counts, made the downtime and recovery longer than necessary, and the response could have drawn better on Pinterest's and AWS's people. The new escalation workflow and monthly backup tests would not have prevented this failure but would speed up the response to a future one.
Sources
Related cases
IL-0379InfluxDB Cloud queries in Azure West Europe slowed and timed out as storage pods fell behind and kept restartingInfluxData · 2026same root cause · shared Database replication, Database
IL-0309Preset's RDS PostgreSQL 14.7 to 17.9 upgrade in US East 1 ran out of storage and took Superset down for six hoursPreset · 2026same root cause · shared Database, Database backups and snapshots
IL-0288ZooKeeperWait rose to thousands of seconds on three redundant ClickHouse instances, degrading a CDN77 data pipelineCDN77 · 2026same root cause · shared Database replication, Database
IL-0305Replicas of one Intercom Vitess shard hit 100% CPU and fell up to 2.2 hours behind, serving stale inbox dataIntercom · 2026same root cause · shared Database replication, Database
IL-0443A Repro Aurora MySQL reader restarted unexpectedly and writer insert latency rose about sevenfold for a day or moreRepro · 2019same root cause · shared Database replication, Database
IL-0450Oracle RAC commits behind order APIs kept waiting over 0.1 s on log file sync while average host CPU had headroomToss Securities · 2026same root cause · shared Database
IL-0553CircleCI's orchestration database filled its transaction log volume, and no workflows or jobs started for 50 minutesCircleCI · 2026same root cause · shared Database
IL-0435Error rates rose as Aurora MySQL killed connections and declined queries to avoid OOM while load metrics stayed flatCyberAgent · 2026same root cause · shared Database
IL-0497Buttondown's database ran out of connections twice in one day and requests failed while health checks mostly passedButtondown · 2026same root cause · shared Database
IL-0350Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast tableIONOS Cloud · 2026same root cause · shared Database
IL-0390PostHog US Cloud event ingestion slowed to delays of up to 2 days as writes to the Persons Postgres table stalledPostHog · 2025same root cause · shared Database
IL-0100PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared Database
IL-0287One overloaded shard of MongoDB's internal monitoring database delayed Atlas metrics, alerts and cluster operationsMongoDB · 2025same root cause · shared Database
IL-0557Database transaction locks piled up on Storj's US1 satellite, and uploads and downloads failed with timeouts for 2 hoursStorj · 2025same root cause · shared Database
IL-0551Bitbucket Cloud's overloaded primary database failed over, and the website, API and Pipelines were slow for almost a dayAtlassian · 2025same root cause · shared Database
IL-0298Wikimedia edits failed for 38 minutes when sessionstore Cassandra nodes in both data centres crashed on full disksWikimedia Foundation · 2025same root cause · shared Database
IL-0312Gadget's main Postgres did not start again after a restart to raise max_connections, and the platform went downGadget · 2024same root cause · shared Database
IL-0544Wasabi's database cache stopped responding when its queueing service filled up, and S3 failed in all regionsWasabi · 2024same root cause · shared Database
IL-0447An every RDS for MySQL instance slowed badly when its gp2 EBS BurstBalance ran out and IOPS fell to the baselineevery · 2024same root cause · shared Database
IL-0375MongoDB Atlas cluster changes and alerts lagged for hours after a control plane metadata migration congested a databaseMongoDB · 2024same root cause · shared Database
IL-0110A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Database
IL-0550Bitbucket Cloud's database slowed until connection pools saturated and requests timed out, leaving Pipelines pendingAtlassian · 2024same root cause · shared Database
IL-0270Discord's ScyllaDB authentication cluster overloaded during an OS upgrade, and logins failed for about 50 minutesDiscord · 2023same root cause · shared Database
IL-0202A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1same root cause · shared Database
IL-0386After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodesincident.io · 2023same root cause · shared Database
IL-0326A retention clean-up job filled the disk of MailPace's managed Postgres, which stayed offline for just over two hoursMailPace · 2023same root cause · shared Database
IL-0546Buildkite jobs could not go to newly connected agents once agent IDs passed a 32-bit integer foreign key's limitBuildkite · 2023same root cause · shared Database
IL-0453Hakuna API latency spiked every day during the ElastiCache Redis backup as memory filled and the backup went forklessHyperconnect · 2022same root cause · shared Database backups and snapshots
IL-0500UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up2i2c · 2022same root cause · shared Database
IL-0301Basecamp 3's primary database went into total gridlock at the top of the hour and Basecamp 3 went down twice37signals · 2022same root cause · shared Database
IL-0206Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022same root cause · shared Database
IL-0302Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022GitHub · 2022same root cause · shared Database
IL-0212Many Slack users could not connect because a Vitess keyspace for channel membership was overloaded by cache-miss queriesSlack · 2022same root cause · shared Database
IL-0456Inflearn went down six times during a January 2022 free-course event as slow queries piled up on Aurora PostgreSQLInflab · 2022same root cause · shared Database
IL-0321GCP snapshots on all Patroni nodes stalled GitLab.com's primary PostgreSQL disk, Patroni failed over and 5xx spikedGitLab · 2021same root cause · shared Database
IL-0496Ably's Cassandra hit 100% CPU in every region and queries timed out globally, and the load returned on two more eveningsAbly · 2020same root cause · shared Database
IL-0037Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instancesDatadog · 2019same root cause · shared Database
IL-0297Mezmo users could not sign in and log indexing lagged after the auth cache's Redis servers were OOM killedMezmo · 2019same root cause · shared Database
IL-0211A Mandrill PostgreSQL shard shut down at transaction ID wraparound and email sending was impaired for about 40 hoursMailchimp · 2019same root cause · shared Database
IL-0064CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019same root cause · shared Database
IL-0300Basecamp 3 went read-only for almost five hours when its events table reached the ceiling of 2,147,483,64737signals · 2018same root cause · shared Database
IL-0333A downsized RDS PostgreSQL database maxed out its CPU at US peak and failing ELB health checks took buildkite.com downBuildkite · 2016same root cause · shared Database
IL-0296Transloadit's us-east Redis cluster lost queued jobs and heartbeats, raising WORKER_JOB_ERROR errors for 81 customersTransloadit · 2016same root cause · shared Database replication
IL-0282DynamoDB in US-East reached about 55% request errors as storage servers overloaded its metadata serviceAmazon Web Services · 2015same root cause · shared Database
IL-0383PagerDuty's shared Cassandra cluster degraded during a repair, delaying notifications and losing incoming eventsPagerDuty · 2014 · sev1same root cause · shared Database
IL-0320Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayedPostmark · 2011same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0334
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0334/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.