Cases5 cases

1 to 5 of 5

1. IL-0510 MIT Learn's RDS PostgreSQL ran out of storage after a release and learn.mit.edu returned 503s and 504s for hours

   MIT Open Learning2026

   Symptoms

   On 24 March 2026 learn.mit.edu went down shortly after a regular release. An engineer reported the outage with a linked Sentry issue, and Sentry soon showed many 504 responses from the site.

   Service outageNode disk fullDatabase overloadLock contention or deadlock

   Trigger

   A regular release deployed a query pattern change to the learning resource APIs that put an aggregation inside the main query.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   Inefficient queryUnrepresentative stagingHarmful manual action

   Detection

   An engineer reported the outage with a linked Sentry issue, and Sentry showed many 504 responses. The RDS FreeStorageSpace metric showed that the database had used up its storage.

   Remediation

   • Rolled back the release's query pattern change
   • Deleted the read replica so the database could flush its buffered WAL files
   • Terminated all active queries running longer than 10 minutes with pg_terminate_backend, after the backup finished and with guidance from AWS Support

   Action items

   • Explore enabling enhanced analytics on the production Learn database
   • Document rollback steps on the platform engineering documentation site
   • Make easy rollback part of the new release management framework
   • Alert on DiskQueueDepth for production databases without making the alert too noisy
   • Alert when API and site response times are more than 30% above normal
   • Run a gameday on an example production database outage
   • Adopt Rootly for incident management, with an automatic channel and an automatic incident transcript
   • Write an outage runbook: start a call, have the on-call engineer watch metrics until they return to baseline, and give one person the communications
   • Put the query that terminates long-running queries, and a version that finds them for selective termination, into a production Postgres runbook
   • Open an issue on redesigning the Learn Resource View Event model, which is only used for counts
   • Use anonymised production data generation tools to bring RC up to production data volume, on separate infrastructure for load testing
   • Change the view count to use a precomputed column

   Lessons

   The change was reviewed by senior engineers and showed no problems locally or on RC, but data scale and load in production are very different, and nothing in the system's health showed that adding a COUNT aggregation would make disk usage explode.

   View case
2. IL-0317 Clerk's Postgres slowed after an automatic analyze flipped a query plan, and over 95% of requests got 429

   Clerk2026

   Symptoms

   At 8:11 AM PST (16:11 UTC) on 19 February 2026, Clerk's database performance dropped at once. Slow database queries made request handlers queue, and over 95% of traffic was answered with 429 without being handled; the small share of requests that reached the database returned 200, but extremely slowly.

   Service outageDatabase overloadHigh latency

   Trigger

   A routine automatic ANALYZE on one table sampled only NULL values in a column, and the Postgres planner switched a frequently run query to an inefficient plan.

   Root cause

   Query plan regression

   Component

   DatabaseQuery planner and table statistics

   Contributing factor

   Inefficient queryMonitoring blind spot

   Detection

   The incident was reported and escalated internally at 16:15 UTC, and the team started a group call.

   Remediation

   • Enabled a new failover that generates session tokens outside the core Session API
   • Re-ran ANALYZE manually on the affected table, which put the query back on its earlier plan
   • Disabled the failover once the database was back to healthy levels
   • Increased the statistics target on the table so the planner samples more rows
   • Refactored the query so the planner could plan it deterministically

   Action items

   • Add dedicated alerting for database query plan flips
   • Train the reliability team to enable the new session token failover and add automatic triggers for it
   • Audit all queries for others at risk of inefficient plans from unstable statistics
   • Name a dedicated communications lead during incidents
   • Post status page updates at a regular cadence even when nothing has changed
   • Make status page severity labels match the impact on customers
   • Cross-post incident updates to social channels

   Lessons

   Clerk says a query plan flip can cause sudden, severe degradation and needs to be detected directly instead of through downstream symptoms; relying on those symptoms led the team to blame a customer's traffic pattern. Customers also told Clerk that status page updates came too rarely, the first update came too late and the initial severity label did not match the impact.

   View case
3. IL-0567 Aurora MySQL behind Cookie Run: OvenBreak hit 100% CPU after a sales-ranking feature launch, and players could not play

   Devsisters2023

   Symptoms

   After further bug-fix deployments and reconnecting users on the launch day of the package sales ranking feature, a database alarm fired and the CPU of the Cookie Run: OvenBreak AWS Aurora MySQL database was at 100%. Database connections then failed, and players could not log in to or play the game.

   Database overloadService outage

   Trigger

   The package sales ranking feature was deployed, and further bug-fix deployments followed that reconnected users; each server update left more servers with empty caches that ran the ranking query against the database.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   Inefficient queryDependence on a warm cacheUnrepresentative staging

   Detection

   A database alarm fired, and the CPU graph of the Aurora MySQL database showed 100%.

   Remediation

   • Put the game servers into maintenance
   • Added a switch for the package sales ranking feature and turned the feature off, which restored the service
   • Rewrote the ranking query to group products that share a sale start time and to drop the current-time condition
   • Moved the ranking query to a scheduler server that runs it periodically and stores one shared cached result
   • Turned the package sales ranking feature back on after both fixes

   Lessons

   A heavy query combined with per-server caches in a distributed server setup caused a problem nobody expected, so the points likely to cause trouble need to be found in advance. During an outage, restoring the service comes before finding the cause, and a switch that turns a feature off, built in ahead of time, is a safety device that helps run a stable service. The online ALTER itself went well with Percona Toolkit and was used again for later schema changes.

   View case
4. IL-0326 A retention clean-up job filled the disk of MailPace's managed Postgres, which stayed offline for just over two hours

   MailPace2023

   Symptoms

   At 13:36 BST on Friday August 11th 2023, minutes after MailPace reran its data clean-up job with more memory, notifications arrived that all database transactions were failing across multiple services. The managed Postgres database had gone offline, did not come back when restarted, and reported that it was out of space.

   Node disk fullService outageWrites rejected

   Trigger

   The daily retention clean-up job, which had been failing with killed, was rerun with its memory raised from 1gb to 8gb, and its large destroy_all deletions filled the database disk.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   Inefficient queryMonitoring blind spotNo headroom

   Detection

   Notifications that all database transactions were failing across multiple services arrived as soon as the outage began; the provider was automatically notified at 99% storage. MailPace's two status pages did not detect the outage.

   Remediation

   • Contacted the managed database provider's support
   • Notified users on Twitter and Mastodon
   • Prepared an offsite backup from six hours earlier on a new database instance as a fallback
   • The provider migrated the database to a larger instance
   • Removed the old data in production by hand, day by day, with an updated query

   Action items

   • Delete old data in batches of 50 records with in_batches in the clean-up job
   • Run the clean-up job four times a day instead of once
   • Stay on a much larger database instance with room to grow
   • Add cron monitoring to all cron jobs
   • Point the status pages at a /health endpoint that checks database access and returns a 500 error on failure
   • Investigate a hot-replica database for automated failover

   Lessons

   Several causes came together: a clean-up job that failed silently, a naive deletion that grew huge once data had piled up, a database plan without headroom for growth, and status pages that missed the outage, presumably because https://app.mailpace.com still answered 200 while the database was down. MailPace also advises that critical services such as email sending be set up with more than one provider.

   View case
5. IL-0456 Inflearn went down six times during a January 2022 free-course event as slow queries piled up on Aurora PostgreSQL

   Inflab2022

   Symptoms

   During a 100% discount event that made 125 courses free, Inflearn received 5 to 20 times its normal traffic, and on four days, 3, 4, 7 and 17 January 2022, the service became unusable around 9 a.m. and 10 p.m. There were six outages in all.

   Service outageDatabase overloadHigh latency

   Trigger

   A 100% discount event on 125 courses, from 3 to 17 January and 18 to 24 January 2022, went viral and brought 5 to 20 times the usual traffic.

   Root cause

   Capacity exhaustion

   Component

   DatabasePostgreSQL vacuum

   Contributing factor

   Inefficient queryMonitoring blind spotExcessive blast radius

   Detection

   The team followed the outages through hourly transaction counts, RDS metrics, monitoring and slow query alerts.

   Remediation

   • Added an index to the interested-companies query on the course page
   • Changed how often the cache data stored in the database was reset and regenerated
   • Moved the read queries that loaded the writer DB to the reader DBs
   • Scaled the database to 8 r5.8xlarge instances, which bought only a few tens of minutes
   • Rewrote slow course queries or added indexes so they use indexes, and switched off features whose queries could not be fixed
   • Removed the large course description column from the select * course queries
   • Split the IDs of the IN clause into batches of 20 run with Promise.all
   • Upgraded PostgreSQL from 10.16 to 11.13
   • Set a 5-second query timeout
   • Raised max connections
   • Ran VACUUM by hand on the course table
   • Moved the large course description column into a separate table

   Action items

   • Change the structure in which one table's failure spreads to the whole service
   • Improve the table and cache structures that are hard to tune
   • Make the SQL code easier to follow
   • Improve code that gives no confidence to change, such as code without types or test code
   • Build a stable batch environment
   • Keep a monolith of reasonable size and separate the domains through which failures spread most

   Lessons

   Because every database bottleneck appeared as slow queries on the reader DBs, watching only the readers hid the root cause on the writer. In PostgreSQL, frequent updates to a table with large columns produce large dead tuples in bulk and slow every query on that table. Before the second event the team could not run a meaningful performance test because the first week showed that almost any page could fail under load. The event exposed high sensitivity to failure in every area, one table's failure spreading to the whole service, hard-to-tune table and cache structures, unintuitive SQL code, code without types or tests, no stable batch environment, and many hidden failure points not yet found.

   View case

By root cause3