Cases5 cases

1 to 5 of 5

1. IL-0309 Preset's RDS PostgreSQL 14.7 to 17.9 upgrade in US East 1 ran out of storage and took Superset down for six hours

   Preset2026

   Symptoms

   A planned major version upgrade of Preset's production RDS PostgreSQL instances in US East 1, from 14.7 to 17.9, ran far past its maintenance window, and all customers on the us2a deployment could not reach Superset. After more than five hours the upgrade failed with "Postgres cluster is in a state where pg_upgrade cannot be completed successfully."

   Service outageUpgrade stall

   Trigger

   A planned major version upgrade of all RDS instances in production US East 1 from PostgreSQL 14.7 to 17.9, started with a ModifyDBInstance API call at 04:12 EDT.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase backups and snapshots

   Contributing factor

   Unrepresentative stagingUnverified assumptionUnknown limitLimited debug access

   Detection

   The planned upgrade was being watched as it ran: the snapshot's I/O was observed at 05:28 EDT, the incident was posted on the status page at 06:00 EDT and an AWS support ticket was opened at 07:26 EDT.

   Remediation

   • AWS rolled the failed upgrade back automatically, and the instance restarted on PostgreSQL 14.15 at 10:13 EDT

   Action items

   • Rehearse the upgrade against a full production-scale dataset to check storage and timing estimates before any future major version upgrade

   Lessons

   Preset learned that RDS takes a full, non-incremental snapshot before a major version upgrade, so the duration of the daily incremental snapshots and of the upgrades in other regions said nothing about how long the pre-upgrade snapshot would take on this instance. The S3 snapshot throughput limit of about 30–45MB/s only becomes visible during such full snapshots, which are rare in normal operation.

   View case
2. IL-0318 Clerk's APIs went down with lock contention and 429s while a Google Cloud SQL live migration slowed its database disk

   Clerk2026

   Symptoms

   At 15:57 UTC on 10 March 2026, Clerk's alerting reported increased latency and elevated 5xx errors across its APIs, and the incident was declared. Monitoring showed increased lock contention in the database, which had started about a minute earlier.

   High latencyHigh error rateLock contention or deadlockService outage

   Trigger

   A routine live migration of the Google Cloud SQL virtual machine that hosts Clerk's Postgres database, done without advance notice to Clerk, did not work as intended.

   Root cause

   Cloud provider outage

   Component

   Database

   Contributing factor

   Limited debug access

   Detection

   Alerting reported increased latency and elevated 5xx errors across the APIs at 15:57 UTC, and monitoring showed lock contention in the database that had begun about a minute earlier.

   Remediation

   • Enabled Origin Outage Mode to keep session management working
   • Opened a ticket with GCP after seeing elevated IO wait on reads
   • Disabled Origin Outage Mode after the live migration completed and the system stabilised

   Action items

   • Ask Google to pin the database again to stop further live migrations
   • Use replica promotion instead of live migrations for all Postgres database maintenance
   • Move to another database provider or run Postgres in-house unless Google offers a way to disable live migrations permanently
   • Return 500 instead of the internal service's 429 so applications can tell the state of Clerk's systems from the error code
   • Shift most of the engineering team to reliability-focused projects

   Lessons

   Clerk concludes that Cloud SQL live migrations are still unsafe after months of joint work with Google, and that they are opaque, leaving Clerk to trust Google's word. Clerk prefers traditional replica promotion for all Postgres maintenance, an automated process it had already used without incident for a Postgres major version upgrade on January 17, 2026.

   View case
3. IL-0315 Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connection

   Railway2025

   Symptoms

   At 18:36 UTC on 28 October 2025, two minutes after a database change went live in production, Railway's internal monitoring alerted and paged the engineering team. The dashboard could not be opened, CLI operations that use the API, such as railway up, failed, GitHub-based deployments were delayed because webhook processing failed, and all Railway Public API operations were disrupted.

   Service outageLock contention or deadlockDatabase overload

   Trigger

   A routine schema change to the backend's Postgres database, introduced at 18:23 UTC and live in production at 18:34 UTC, that added a new column with an index without the CONCURRENTLY option.

   Root cause

   Faulty deployment

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   No configuration testsLimited debug access

   Detection

   Internal monitoring alerts paged the engineering team at 18:36 UTC, and the incident was declared at 18:41 UTC.

   Remediation

   • The index migration finished after about 30 minutes and released the table lock, and the queued operations and connection pools recovered

   Action items

   • Enforce CONCURRENTLY for all index creation in CI and block pull requests that do not use it
   • Adjust PgBouncer pool limits so they cannot overwhelm the Postgres instance's capacity
   • Configure database user connection limits so administrative access stays possible during incidents

   View case
4. IL-0508 All SourceHut services went down when the main database server remilia.sr.ht failed in hardware and became unreachable

   SourceHut2023

   Symptoms

   At about 13:40 UTC on 11 January 2023, alerts started firing for most SourceHut services being unavailable, along with one for remilia.sr.ht, the physical host that is SourceHut's main database server. The database server could not be reached remotely at all.

   Service outage

   Trigger

   remilia.sr.ht, the main database server, suffered a hardware failure at about 13:38 UTC.

   Root cause

   Hardware failure

   Component

   Database

   Contributing factor

   Single point of failureLimited debug access

   Detection

   Alerts fired at about 13:40 UTC for most services and for remilia.sr.ht itself, after Prometheus started failing to scrape remilia at 13:38.

   Remediation

   • Sent SourceHut's emergency on-call contact in the datacenter area to go on site
   • Moved remilia's disks into the chassis of build worker cirno1.sr.ht and brought it online as remilia

   Action items

   • Make the off-site database copy able to serve as a failover database host
   • Re-prioritize the migration to the EU
   • Set up new database failover
   • Add a new EU build worker to backfill build capacity
   • Record a better hardware inventory for future reference

   View case
5. IL-0334 Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went down

   Instapaper2017

   Symptoms

   On Wednesday, February 9, 2017 at 12:30PM PT, inserts of new entries into the bookmarks table of Instapaper's MySQL database on Amazon RDS started to fail with an error. The bookmarks table, which stores the articles Instapaper users save, had grown past 2TB, and the Instapaper service was down until Thursday, February 10 at 7:30pm PT.

   Writes rejectedService outage

   Trigger

   At 12:30PM PT on February 9 the bookmarks table grew past the 2TB file size limit of the instance's filesystem.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase backups and snapshotsDatabase replication

   Contributing factor

   Unknown limitMonitoring blind spotUntested backupNo runbookLimited debug access

   Remediation

   • Dumped the production database to a 5.5TB RAID-0 disk on an i2.8xlarge instance with Pinterest's SRE team
   • Launched a temporary production instance with limited access to archives
   • Set up an Aurora read replica of the old database as one recovery path
   • Imported the data into a new MySQL RDS instance without secondary indexes, then built the indexes
   • Had an Amazon engineer rsync the data from the ext3 filesystem to an ext4 filesystem mounted on the failed database
   • Synced the new ext4 database with the temporary production database through row-based replication
   • Promoted the new database to master and pointed the application at it

   Action items

   • Define a workflow that escalates system-wide Instapaper outages immediately to Pinterest's SRE team
   • Test MySQL backups every month instead of every three months
   • Put a process in place with Pinterest's SRE team to recover from this type of failure

   Lessons

   A filesystem limit that Instapaper did not know about and could not see made both the production database and all its backups useless, and the only way out was to restore the data onto a new filesystem. Because the hosted database could only be reached through MySQL, filesystem-level fixes such as rsync needed Amazon engineers. Even with a perfect response, rebuilding the database would have taken at least 10 hours. The lack of a disaster recovery plan for this kind of failure, and the estimate of the dump time from row counts, made the downtime and recovery longer than necessary, and the response could have drawn better on Pinterest's and AWS's people. The new escalation workflow and monthly backup tests would not have prevented this failure but would speed up the response to a future one.

   View case

By root cause4