Cases20 cases

1 to 5 of 20

1. IL-0570 Dropping an unused MySQL table stalled queries on its busy parent table and crowdworks.jp was hard to reach for a minute

   CrowdWorks2026

   Symptoms

   Right after a deploy ran a migration with drop_table to remove a table that was no longer used, database queries started to pile up and crowdworks.jp was hard to reach for about a minute. The migration finally failed on a timeout and the service recovered on its own.

   Lock contention or deadlockIntermittent connection failures

   Trigger

   An engineer's deploy ran a migration that used drop_table to delete a table no longer in use.

   Root cause

   Faulty deployment

   Component

   DatabaseSchema migration

   Contributing factor

   –

   Action items

   • In migrations that may take an exclusive MDL on a high-traffic table, through DDL on the table, adding or removing a foreign key to it, or creating or dropping a table with such a foreign key, run SET SESSION lock_wait_timeout = 3 first
   • Write these migrations with def up and def down instead of def change, setting the timeout before the DDL in both
   • Put only one such DDL in each migration
   • Release migrations that contain such DDL on their own
   • Run them during low-traffic hours
   • Document the rules in db/migrate/README.md
   • Point AGENTS.md at that README so Copilot and other AI agents review migrations against it
   • Add the check to the automatic comment posted on pull requests that change db/migrate

   View case
2. IL-0322 A GCP storage incident degraded the disk under GitLab.com's primary PostgreSQL and caused two bursts of site-wide 503s

   GitLab2026sev1

   Symptoms

   On 2026-06-11 https://gitlab.com returned widespread, intermittent HTTP 503 errors across all major services in two bursts of about 5 to 10 minutes, 06:16 to 06:22 UTC and 07:03 to 07:12 UTC. A Pingdom alert reported the first burst in triage at 06:16:12, about four minutes after impact began at 06:12, and each burst ended on its own.

   Service outageIntermittent timeoutsConnection pool exhaustionLock contention or deadlockReplication lag

   Trigger

   A GCP storage incident degraded the IO of a disk in the `us-east1-b` zone that backed GitLab.com's primary PostgreSQL node.

   Root cause

   Cloud provider outage

   Component

   DatabaseCloud block storage

   Contributing factor

   Single point of failureNo alert on limitNo runbook

   Detection

   A Pingdom alert on the user-facing 503 errors was reported in triage at 06:16:12 UTC. No alert existed for disk IO saturation, pgBouncer pool saturation or LWLock contention.

   Remediation

   • Escalated to Database Operations experts
   • Paused deployments
   • Opened a GCP support case and engaged the GCP TAM, and GCP confirmed the storage incident
   • Prepared a Patroni primary switchover to node 04 in us-east1-d as a contingency, without executing it
   • Held a 3-hour pause-and-review window and resolved the incident after no recurrence

   Action items

   • Add disk IO saturation and queue-depth alerting for primary Patroni database nodes
   • Write a Patroni primary switchover runbook with explicit trigger thresholds
   • Clarify incident state-transition guidance for intermittent, self-resolving issues
   • Decide whether the primary should fail over automatically during disk IO degradation
   • Find a short-term mitigation that limits the blast radius of a single degraded disk until Cells do so
   • Make application database load balancing fail fast or reroute when the primary stops responding
   • Consider alerting on pgBouncer pool saturation and LWLock contention as leading indicators

   Lessons

   A single degraded disk was enough to take down GitLab.com because all primary database traffic goes through one node and requests waited for the Rack timeout instead of failing fast. The leading indicators, disk IO queue depth, pgBouncer pool saturation and LWLock contention, rose before full impact but had no alerts. Without decision criteria for a switchover, responders spent much of an intermittent incident waiting, and it was hard to decide when to move the incident to Fixing or Monitoring.

   View case
3. IL-0510 MIT Learn's RDS PostgreSQL ran out of storage after a release and learn.mit.edu returned 503s and 504s for hours

   MIT Open Learning2026

   Symptoms

   On 24 March 2026 learn.mit.edu went down shortly after a regular release. An engineer reported the outage with a linked Sentry issue, and Sentry soon showed many 504 responses from the site.

   Service outageNode disk fullDatabase overloadLock contention or deadlock

   Trigger

   A regular release deployed a query pattern change to the learning resource APIs that put an aggregation inside the main query.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   Inefficient queryUnrepresentative stagingHarmful manual action

   Detection

   An engineer reported the outage with a linked Sentry issue, and Sentry showed many 504 responses. The RDS FreeStorageSpace metric showed that the database had used up its storage.

   Remediation

   • Rolled back the release's query pattern change
   • Deleted the read replica so the database could flush its buffered WAL files
   • Terminated all active queries running longer than 10 minutes with pg_terminate_backend, after the backup finished and with guidance from AWS Support

   Action items

   • Explore enabling enhanced analytics on the production Learn database
   • Document rollback steps on the platform engineering documentation site
   • Make easy rollback part of the new release management framework
   • Alert on DiskQueueDepth for production databases without making the alert too noisy
   • Alert when API and site response times are more than 30% above normal
   • Run a gameday on an example production database outage
   • Adopt Rootly for incident management, with an automatic channel and an automatic incident transcript
   • Write an outage runbook: start a call, have the on-call engineer watch metrics until they return to baseline, and give one person the communications
   • Put the query that terminates long-running queries, and a version that finds them for selective termination, into a production Postgres runbook
   • Open an issue on redesigning the Learn Resource View Event model, which is only used for counts
   • Use anonymised production data generation tools to bring RC up to production data volume, on separate infrastructure for load testing
   • Change the view count to use a precomputed column

   Lessons

   The change was reviewed by senior engineers and showed no problems locally or on RC, but data scale and load in production are very different, and nothing in the system's health showed that adding a COUNT aggregation would make disk usage explode.

   View case
4. IL-0487 Autumn's Postgres CPU spiked and about 10% of API requests failed after a collation change left the main query unindexed

   Autumn2026

   Symptoms

   On Sunday March 15, 2026 at about 15:00 GMT, right after collation changes were applied to the customer_entitlements and customer_products tables, database CPU on Autumn's production Postgres spiked and queries began to fail. About 10% of API requests to Autumn failed until 16:15 GMT.

   Database overloadHigh error rateLock contention or deadlock

   Trigger

   A migration that aligned column collations across related tables changed the collation of customer_entitlements and then customer_products on the production database.

   Root cause

   Configuration change

   Component

   Database

   Contributing factor

   Ineffective rollback

   Remediation

   • Shut down all application connections to the database
   • Reverted the collation changes
   • Ran ANALYZE to rebuild the planner statistics and restarted

   Action items

   • Do all future column and schema migrations with the duplicate-dual-write-cutover pattern: add a new column, dual-write, backfill in small batches, create indexes concurrently, then cut over
   • Add a status page
   • Consider making the check and track SDKs fail open by default

   View case
5. IL-0318 Clerk's APIs went down with lock contention and 429s while a Google Cloud SQL live migration slowed its database disk

   Clerk2026

   Symptoms

   At 15:57 UTC on 10 March 2026, Clerk's alerting reported increased latency and elevated 5xx errors across its APIs, and the incident was declared. Monitoring showed increased lock contention in the database, which had started about a minute earlier.

   High latencyHigh error rateLock contention or deadlockService outage

   Trigger

   A routine live migration of the Google Cloud SQL virtual machine that hosts Clerk's Postgres database, done without advance notice to Clerk, did not work as intended.

   Root cause

   Cloud provider outage

   Component

   Database

   Contributing factor

   Limited debug access

   Detection

   Alerting reported increased latency and elevated 5xx errors across the APIs at 15:57 UTC, and monitoring showed lock contention in the database that had begun about a minute earlier.

   Remediation

   • Enabled Origin Outage Mode to keep session management working
   • Opened a ticket with GCP after seeing elevated IO wait on reads
   • Disabled Origin Outage Mode after the live migration completed and the system stabilised

   Action items

   • Ask Google to pin the database again to stop further live migrations
   • Use replica promotion instead of live migrations for all Postgres database maintenance
   • Move to another database provider or run Postgres in-house unless Google offers a way to disable live migrations permanently
   • Return 500 instead of the internal service's 429 so applications can tell the state of Clerk's systems from the error code
   • Shift most of the engineering team to reliability-focused projects

   Lessons

   Clerk concludes that Cloud SQL live migrations are still unsafe after months of joint work with Google, and that they are opaque, leaving Clerk to trust Google's word. Clerk prefers traditional replica promotion for all Postgres maintenance, an automated process it had already used without incident for a Postgres major version upgrade on January 17, 2026.

   View case

By root cause8