IL-0510 MIT Open Learning · 2026-03-24databaseaws
MIT Learn's RDS PostgreSQL ran out of storage after a release and learn.mit.edu returned 503s and 504s for hours
The learning resource APIs of MIT Learn were slow, especially the featured resources API, and Sentry pointed at a large query that fetched all view events to count them. An engineer changed the query pattern so that the main query contained an aggregation. The team's two most senior engineers reviewed the change, it showed no issues locally or on the RC environment, and it went out in a regular release. DRF computes the count field of a paginated response by running SELECT COUNT * ($QUERY) over the whole query; with the aggregation inside, this worked on smaller datasets but not on production data. On the RDS PostgreSQL instance ol-mitlearn-db-production in us-east-1, the query thrashed temporary files, filled the storage and backed up the database's internal work so that new transactions could not be processed.
The change was rolled back and the database was declared up at 4:31, but the site returned 503 errors again at 4:51. The team deleted the read replica so the database could flush buffered WAL files, dropped the replication slot and ran VACUUM ANALYZE, without bringing DiskQueueDepth down. After an RDS backup finished, and with help from AWS Support, the team terminated every active query that had been running for more than 10 minutes with pg_terminate_backend. Free storage and DiskQueueDepth then returned to normal around 6:24 PM. The write-up gives the total outage as 3 hours and 35 seconds.
Afterwards the team changed the view count to use a precomputed column, and response times of the affected APIs went from ~13-15s to 150-200ms. AWS Support recommended testing ORM query changes with production-like data volumes and EXPLAIN ANALYZE, setting temp_file_limit and log_temp_files, and considering RDS Optimized Reads, which place temporary files on local NVMe storage instead of EBS.
- Severity
- –
- Scope
- learn.mit.edu returned 503 and 504 errors while the production database was out of storage
- Users affected
- –
- Region
- us-east-1
- Time to detect
- –
- Time to resolve
- 3h
Symptoms
On 24 March 2026 learn.mit.edu went down shortly after a regular release. An engineer reported the outage with a linked Sentry issue, and Sentry soon showed many 504 responses from the site.
The MIT Learn production database had used up all of its usable storage, and the growth had started at 18:35 UTC. The PostgreSQL server was thrashing its temporary filesystem. The RDS FreeStorageSpace metric dropped, and later the CheckpointLag and DiskQueueDepth metrics looked unhealthy; the team treats DiskQueueDepth as the critical health metric for RDS.
The first suspects were the Qdrant cluster and a pgdump snapshot an engineer had just created, and both were ruled out within about ten minutes. The engineer who wrote the release's query pattern change then identified it as the cause. The database was declared up and stable at 4:31, but the site still returned 503 errors at 4:51, and when the site was reported back at 5:26, DiskQueueDepth was still unhealthy.
Deleting the read replica so the database could flush its buffered WAL files, dropping the replication slot and running VACUUM ANALYZE did not bring DiskQueueDepth down. Dropping the slot had minimal effect, and VACUUM ANALYZE started a number of additional queries and may have made the problem worse.
Symptom Service outageNode disk fullDatabase overloadLock contention or deadlock
Root cause
Root cause Faulty deployment
- Trigger
- A regular release deployed a query pattern change to the learning resource APIs that put an aggregation inside the main query.
Detection
An engineer reported the outage with a linked Sentry issue, and Sentry showed many 504 responses. The RDS FreeStorageSpace metric showed that the database had used up its storage.
Remediation
- Rolled back the release's query pattern change
- Deleted the read replica so the database could flush its buffered WAL files
- Terminated all active queries running longer than 10 minutes with pg_terminate_backend, after the backup finished and with guidance from AWS Support
Action items
- Explore enabling enhanced analytics on the production Learn database
- Document rollback steps on the platform engineering documentation site
- Make easy rollback part of the new release management framework
- Alert on DiskQueueDepth for production databases without making the alert too noisy
- Alert when API and site response times are more than 30% above normal
- Run a gameday on an example production database outage
- Adopt Rootly for incident management, with an automatic channel and an automatic incident transcript
- Write an outage runbook: start a call, have the on-call engineer watch metrics until they return to baseline, and give one person the communications
- Put the query that terminates long-running queries, and a version that finds them for selective termination, into a production Postgres runbook
- Open an issue on redesigning the Learn Resource View Event model, which is only used for counts
- Use anonymised production data generation tools to bring RC up to production data volume, on separate infrastructure for load testing
- Change the view count to use a precomputed column
Lessons
The change was reviewed by senior engineers and showed no problems locally or on RC, but data scale and load in production are very different, and nothing in the system's health showed that adding a COUNT aggregation would make disk usage explode.
Sources
References
Related cases
IL-0356Every domain hosted on Railway returned HTTP 404 to new connections for five minutes because route lookups failedRailway · 2026same root cause · shared Database
IL-0136A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hoursMicrosoft Azure · 2026same root cause · shared Database
IL-0570Dropping an unused MySQL table stalled queries on its busy parent table and crowdworks.jp was hard to reach for a minuteCrowdWorks · 2026same root cause · shared Database
IL-0380A compaction change in InfluxDB Cloud us-east-1 dropped data from compacted files, and queries missed data for weeksInfluxData · 2026same root cause · shared Database
IL-0369Railway's backend API went down when a column migration waited on locks and replicas could not get database connectionsRailway · 2025same root cause · shared Database
IL-0547Buildkite annotation queries timed out and drove database CPU up across shards after a migration removed an indexBuildkite · 2025same root cause · shared Database
IL-0315Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connectionRailway · 2025same root cause · shared Database
IL-0327After a blue/green upgrade of CircleCI's workflows database every query hit disk, and workflow starts lagged or failedCircleCI · 2025same root cause · shared Database
IL-0324Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database clusterWikimedia Foundation · 2025same root cause · shared Database
IL-0307Supabase's Postgres connection pooler Supavisor 2.0 dropped connections for about 80% of us-east-1 tenantsSupabase · 2025same root cause · shared Database
IL-0552CircleCI's archival database ran out of storage, slowing workflow and job starts and the display of workflows in the UICircleCI · 2025same root cause · shared Database
IL-0554A database migration used up Sentry US's Postgres connections, and the site and event ingestion were down for 22 minutesSentry · 2024same root cause · shared Database
IL-0571STORES reservation returned bursts of 504 errors when a smart-list query held a database process for over 20 minutesSTORES · 2024same root cause · shared Database
IL-0293Cloud Bigtable in us-central1 had higher latency and connection errors for about 32 hours from low-rate packet lossGoogle Cloud · 2023same root cause · shared Database
IL-0523TANF-app production deployments were blocked for two months when a search-index migration ran out of database disk spaceU.S. Department of Health and Human Services · 2023same root cause · shared Database
IL-0567Aurora MySQL behind Cookie Run: OvenBreak hit 100% CPU after a sales-ranking feature launch, and players could not playDevsisters · 2023same root cause · shared Database
IL-0483Replicate's website and API returned slow responses and HTTP 500s for two hours as PostgreSQL connections ran outReplicate · 2023same root cause · shared Database
IL-0371Half of new connections to Render Postgres databases with point-in-time recovery failed while their backup jobs ranRender · 2023same root cause · shared Database
IL-0524An index migration locked Open Collective's whole Transactions table and degraded the service for nearly an hourOpen Collective · 2023same root cause · shared Database
IL-0328A field type change in the database of CircleCI's job distribution service left all jobs stuck in "not running"CircleCI · 2021same root cause · shared Database
IL-0566Hot ranges after a Cookie Run: Kingdom guild update drove CockroachDB nodes to 100% CPU, forcing emergency maintenanceDevsisters · 2021same root cause · shared Database
IL-0391GoCardless API requests timed out for about 15 seconds while a Postgres migration waited for an AccessExclusive lockGoCardless · 2015same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0510
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0510/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.