Cases11 cases

1 to 5 of 11

1. IL-0510 MIT Learn's RDS PostgreSQL ran out of storage after a release and learn.mit.edu returned 503s and 504s for hours

   MIT Open Learning2026

   Symptoms

   On 24 March 2026 learn.mit.edu went down shortly after a regular release. An engineer reported the outage with a linked Sentry issue, and Sentry soon showed many 504 responses from the site.

   Service outageNode disk fullDatabase overloadLock contention or deadlock

   Trigger

   A regular release deployed a query pattern change to the learning resource APIs that put an aggregation inside the main query.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   Inefficient queryUnrepresentative stagingHarmful manual action

   Detection

   An engineer reported the outage with a linked Sentry issue, and Sentry showed many 504 responses. The RDS FreeStorageSpace metric showed that the database had used up its storage.

   Remediation

   • Rolled back the release's query pattern change
   • Deleted the read replica so the database could flush its buffered WAL files
   • Terminated all active queries running longer than 10 minutes with pg_terminate_backend, after the backup finished and with guidance from AWS Support

   Action items

   • Explore enabling enhanced analytics on the production Learn database
   • Document rollback steps on the platform engineering documentation site
   • Make easy rollback part of the new release management framework
   • Alert on DiskQueueDepth for production databases without making the alert too noisy
   • Alert when API and site response times are more than 30% above normal
   • Run a gameday on an example production database outage
   • Adopt Rootly for incident management, with an automatic channel and an automatic incident transcript
   • Write an outage runbook: start a call, have the on-call engineer watch metrics until they return to baseline, and give one person the communications
   • Put the query that terminates long-running queries, and a version that finds them for selective termination, into a production Postgres runbook
   • Open an issue on redesigning the Learn Resource View Event model, which is only used for counts
   • Use anonymised production data generation tools to bring RC up to production data volume, on separate infrastructure for load testing
   • Change the view count to use a precomputed column

   Lessons

   The change was reviewed by senior engineers and showed no problems locally or on RC, but data scale and load in production are very different, and nothing in the system's health showed that adding a COUNT aggregation would make disk usage explode.

   View case
2. IL-0317 Clerk's Postgres slowed after an automatic analyze flipped a query plan, and over 95% of requests got 429

   Clerk2026

   Symptoms

   At 8:11 AM PST (16:11 UTC) on 19 February 2026, Clerk's database performance dropped at once. Slow database queries made request handlers queue, and over 95% of traffic was answered with 429 without being handled; the small share of requests that reached the database returned 200, but extremely slowly.

   Service outageDatabase overloadHigh latency

   Trigger

   A routine automatic ANALYZE on one table sampled only NULL values in a column, and the Postgres planner switched a frequently run query to an inefficient plan.

   Root cause

   Query plan regression

   Component

   DatabaseQuery planner and table statistics

   Contributing factor

   Inefficient queryMonitoring blind spot

   Detection

   The incident was reported and escalated internally at 16:15 UTC, and the team started a group call.

   Remediation

   • Enabled a new failover that generates session tokens outside the core Session API
   • Re-ran ANALYZE manually on the affected table, which put the query back on its earlier plan
   • Disabled the failover once the database was back to healthy levels
   • Increased the statistics target on the table so the planner samples more rows
   • Refactored the query so the planner could plan it deterministically

   Action items

   • Add dedicated alerting for database query plan flips
   • Train the reliability team to enable the new session token failover and add automatic triggers for it
   • Audit all queries for others at risk of inefficient plans from unstable statistics
   • Name a dedicated communications lead during incidents
   • Post status page updates at a regular cadence even when nothing has changed
   • Make status page severity labels match the impact on customers
   • Cross-post incident updates to social channels

   Lessons

   Clerk says a query plan flip can cause sudden, severe degradation and needs to be detected directly instead of through downstream symptoms; relying on those symptoms led the team to blame a customer's traffic pattern. Customers also told Clerk that status page updates came too rarely, the first update came too late and the initial severity label did not match the impact.

   View case
3. IL-0287 One overloaded shard of MongoDB's internal monitoring database delayed Atlas metrics, alerts and cluster operations

   MongoDB2025

   Symptoms

   On 7 October 2025 MongoDB reported a delay in the metric ingestion pipeline of MongoDB Atlas and Cloud Manager. Metrics in Atlas dashboards arrived late or went missing, Host Down alerts did not fire, and Atlas cluster management operations such as provisioning and resizing were delayed, while customer clusters themselves stayed healthy and available.

   Monitoring outageDatabase overloadLoad imbalance

   Trigger

   The rollout of an updated internal software version caused a temporary spike in resource demand on the internal database cluster.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   Inefficient queryNo headroom

   Remediation

   • Took corrective actions to stabilise the system and monitored their effect
   • Implemented a fix and monitored the results until the incident was resolved

   Action items

   • Shard the collections with concentrated load to spread traffic over more nodes
   • Optimise inefficient queries
   • Provision additional infrastructure capacity for higher traffic
   • Refine capacity planning to anticipate future load spikes, using metrics from this event
   • Redesign software rollouts to account for the predictable rise in resource demand during a rollout
   • Keep monitoring how the sharding and query changes resolve the hotspots
   • Expand the detection of elevated load conditions for faster anomaly detection and response

   View case
4. IL-0551 Bitbucket Cloud's overloaded primary database failed over, and the website, API and Pipelines were slow for almost a day

   Atlassian2025

   Symptoms

   At 15:26 UTC on 8 May 2025, the Bitbucket Cloud website and API became slow because the primary database was overloaded. Atlassian's real-time monitoring detected the problem at once.

   Database overloadHigh latencyHigh error rate

   Trigger

   A high-scale backfill job run by an internal Atlassian service sent a large volume of calls to certain Bitbucket API endpoints, which ran expensive queries against the primary database.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   Inefficient queryRetry storm

   Detection

   Atlassian's real-time monitoring detected the incident as soon as it started.

   Remediation

   • Stopped the high-intensity backfill job
   • Primary database failed over automatically, and Bitbucket services recovered within 15 minutes
   • Applied several measures after the failover to shed database load

   Action items

   • Route more reads to read replicas instead of the write primary database
   • Adjust rate limits for internal API endpoints that run resource-intensive database operations
   • Optimise database queries so they run more efficiently
   • Tune the retry policies of downstream services

   View case
5. IL-0571 STORES reservation returned bursts of 504 errors when a smart-list query held a database process for over 20 minutes

   STORES2024

   Symptoms

   About 20 minutes into a batch that rewrote the preset smart-list data of STORES 予約 (STORES reservation), an alert fired for a sudden rise in 504 errors (Gateway Timeout). The work had started a little after 03:00 UTC, and the metrics watched during it, such as web server response latency and the load of the writer database, had shown no large change.

   High error rateDatabase overload

   Trigger

   The preset smart lists 初回予約の顧客 (first-reservation customers) and 要フォロー顧客 (customers needing follow-up), created more than a month earlier, were rewritten with new search parameters after the search specification changed, which switched them onto the new search query.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   Inefficient queryChange with a delayed effectUnrepresentative stagingMonitoring blind spotUnverified assumption

   Detection

   An alert on a sudden increase in 504 errors (Gateway Timeout), about 20 minutes after the preset rewrite batch started.

   Action items

   • Rework the database design so the inline-view search logic is no longer used
   • Add a test that checks this API's EXPLAIN output for scans larger than expected
   • Provide an environment where the timing of this logic can be measured easily

   Lessons

   Rushing to meet the schedule led to both a failure that affected users and a later release. In a multi-tenant database an EXPLAIN that shows index use can still hide large scans, tests need the case where one query is fast and the next is slow, the metrics to watch during a data change are those of the instance that serves the affected queries, and metrics sent at the end of a request are lost for requests that never finish.

   View case

By root cause3