Cases8 cases

3 to 7 of 8

1. IL-0498 Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four days

   Clerk2025

   Symptoms

   At 13:08 UTC on 15 September 2025, Clerk's internal monitoring alerted on significant request failures and increased latency. Over the following days the request failures and latency spikes kept coming back while the Postgres database stayed unhealthy, with repeated spikes in database load.

   High error rateHigh latencyDatabase overloadConnection pool exhaustion

   Trigger

   At 05:30 UTC on 14 September 2025 the cloud provider automatically upgraded Clerk's Postgres database to a new minor version, which changed how Postgres handled connection locks.

   Root cause

   Breaking change in an upgrade

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Unpinned versionRetry stormMonitoring blind spotOverlapping failuresHarmful manual action

   Detection

   Internal monitoring alerted on significant request failures and increased latency at 13:08 UTC on 15 September. Shortly after the upgrade on 14 September it had already shown an increased but still acceptable database load.

   Remediation

   • Optimised several expensive queries, refined critical indexes and re-indexed the database
   • Blocked abusive traffic more aggressively for a time
   • Performed a manual minor version upgrade and increased database capacity
   • Added jitter to the connection pooling configuration so connections no longer recycled in step

   Action items

   • Evaluate other database providers that give more control over upgrades and downgrades
   • Isolate more services from each other, for example Sign Up and Sign In infrastructure from the rest of the system
   • Keep optimising the most costly database queries
   • Evaluate sharding and multi-tenant infrastructure to isolate one application from others

   Lessons

   Clerk's review found that connection metrics sampled every 60 seconds could not show a recycling pattern that lasted seconds and repeated every 15 minutes, that an attack and background jobs with similar periods pulled attention away from the database layer, that the Backend API looked healthy only because it manages far fewer connections, and that the release freeze removed the staggering of connection creation that normal deploys provide.

   View case
2. IL-0571 STORES reservation returned bursts of 504 errors when a smart-list query held a database process for over 20 minutes

   STORES2024

   Symptoms

   About 20 minutes into a batch that rewrote the preset smart-list data of STORES 予約 (STORES reservation), an alert fired for a sudden rise in 504 errors (Gateway Timeout). The work had started a little after 03:00 UTC, and the metrics watched during it, such as web server response latency and the load of the writer database, had shown no large change.

   High error rateDatabase overload

   Trigger

   The preset smart lists 初回予約の顧客 (first-reservation customers) and 要フォロー顧客 (customers needing follow-up), created more than a month earlier, were rewritten with new search parameters after the search specification changed, which switched them onto the new search query.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   Inefficient queryChange with a delayed effectUnrepresentative stagingMonitoring blind spotUnverified assumption

   Detection

   An alert on a sudden increase in 504 errors (Gateway Timeout), about 20 minutes after the preset rewrite batch started.

   Action items

   • Rework the database design so the inline-view search logic is no longer used
   • Add a test that checks this API's EXPLAIN output for scans larger than expected
   • Provide an environment where the timing of this logic can be measured easily

   Lessons

   Rushing to meet the schedule led to both a failure that affected users and a later release. In a multi-tenant database an EXPLAIN that shows index use can still hide large scans, tests need the case where one query is fast and the next is slow, the metrics to watch during a data change are those of the instance that serves the affected queries, and metrics sent at the end of a request are lost for requests that never finish.

   View case
3. IL-0308 Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slow

   RevenueCat2022

   Symptoms

   At 9:34 UTC on November 23, 2022, RevenueCat started moving traffic from its Aurora PostgreSQL 10.x cluster to a new Aurora PostgreSQL 14.x cluster. By 9:39 UTC all traffic was on the new cluster, latency was bad, and the RevenueCat dashboard and the endpoints used by the RevenueCat SDK stopped working.

   Database overloadHigh latencyService outageConnection pool exhaustion

   Trigger

   At 9:34 UTC production traffic was switched from the Aurora PostgreSQL 10.x cluster to the new Aurora PostgreSQL 14.x cluster, before ANALYZE had run on its largest tables.

   Root cause

   Breaking change in an upgrade

   Component

   DatabasePostgreSQL vacuumDatabase replicationDatabase proxy and connection poolerQuery planner and table statistics

   Contributing factor

   Unrepresentative stagingUnverified assumptionAutomation amplificationMonitoring blind spot

   Detection

   At 9:39 UTC, with all traffic on the new cluster, latency looked bad and the infrastructure team began to investigate.

   Remediation

   • Added a statement timeout to the database user so queries could not hold a connection too long
   • Shed 90% of traffic by returning 503s from the load balancer
   • Dropped indexes that the planner was using wrongly and that were not needed
   • Stopped accepting Apple webhooks for a time
   • Rolled out a release without New Relic to stop its EXPLAIN queries
   • Ran ANALYZE by hand on the biggest tables
   • Raised the values of the affected primary key sequences with a prepared script
   • Stopped the integration event dispatcher by hand
   • Replayed the webhooks that were dispatched but not recorded
   • Restored integration event dispatching, ETLs and other non-web server jobs

   Action items

   • Consider change freezes during business-critical periods
   • Explain better how the SDK behaves during backend outages
   • Tell customers about maintenance and outages in the developer dashboard and by direct email to technical contacts
   • Replay a subset of write traffic in migration dry runs to mimic the cutover
   • Move costly processes to asynchronous work to keep the API fast and stable
   • Re-evaluate table indexes, including unused ones the database may choose for queries
   • Show the load of each endpoint and system in the database dashboards
   • Involve support and customer communications staff during outages

   Lessons

   The new cluster had not been analyzed, and the tests missed it because the test cluster had been up long enough to gather statistics. Dry runs replayed no write traffic, on the belief that writes are simpler than reads, and the team assumed sequence state was replicated with the data. Rolling back at the start would have been better than fixing forward. The migration, timed to finish before Thanksgiving and Black Friday, should have been delayed until the week after, because the US had entered a sensitive period and part of the US support team was already on holiday.

   View case
4. IL-0301 Basecamp 3's primary database went into total gridlock at the top of the hour and Basecamp 3 went down twice

   37signals2022

   Symptoms

   At 13:00 UTC on Monday, June 27, 2022, Basecamp 3 became extremely slow, then showed error pages, and then stopped working altogether. Basecamp 2 and Basecamp Classic kept working normally.

   Lock contention or deadlockDatabase overloadHigh latencyHigh error rateService outage

   Trigger

   At 13:00 UTC, when Basecamp 3 sends scheduled notifications in massive batches, a rush of demanding queries reached the primary database and started a cascade of heavy lock contention.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No capacity planningHarmful manual actionMonitoring blind spot

   Detection

   The team saw the Basecamp 3 primary database become completely congested within 30 seconds of 13:00 UTC and responded immediately.

   Remediation

   • Paused and disabled parts of Basecamp 3, such as delaying progress report emails
   • Failed over from the primary database to a replica, which brought Basecamp 3 back until it tipped over again
   • Put Basecamp 3 into maintenance mode
   • Set hard limits on database query time
   • Brought features back online step by step with the query limits in place

   Action items

   • Keep hard limits on database query time so slow queries fail instead of piling up
   • Tune the limits on concurrency
   • Treat top-of-the-hour database load spikes from scheduled batches as unacceptable
   • Reduce the number of parallel workers for some scheduled jobs
   • Let some scheduled activities run over several minutes instead of all at once
   • Move bursty queries from the primary database to read-only replicas
   • Fix the quiet performance regressions found in a review of the past year of database metrics
   • Find and spread out other bursty activities that concentrate load on the database
   • Tune the database for more predictable behaviour under heavy load and high concurrency
   • Add observability into the database's internal locking and transaction states
   • Build tools to switch off high-load Basecamp 3 features selectively during an incident
   • Run operational fire drills for a wedged database
   • Evaluate upgrading the database software and hardware
   • Set service level objectives for the components behind Basecamp 3
   • Hold quarterly performance and load reviews to catch slow, subtle changes

   Lessons

   A system running just past its limits can flip from stable to unstable without any particular trigger, and steady growth plus small regressions that raised no alarm had brought Basecamp 3's primary database to that point. Under lock contention, terminating queries does not relieve the database, because rolling back their transactions waits on the same locks. The team struggled to respond with confidence because there was no single clear cause and little visibility into the database's internals.

   View case
5. IL-0456 Inflearn went down six times during a January 2022 free-course event as slow queries piled up on Aurora PostgreSQL

   Inflab2022

   Symptoms

   During a 100% discount event that made 125 courses free, Inflearn received 5 to 20 times its normal traffic, and on four days, 3, 4, 7 and 17 January 2022, the service became unusable around 9 a.m. and 10 p.m. There were six outages in all.

   Service outageDatabase overloadHigh latency

   Trigger

   A 100% discount event on 125 courses, from 3 to 17 January and 18 to 24 January 2022, went viral and brought 5 to 20 times the usual traffic.

   Root cause

   Capacity exhaustion

   Component

   DatabasePostgreSQL vacuum

   Contributing factor

   Inefficient queryMonitoring blind spotExcessive blast radius

   Detection

   The team followed the outages through hourly transaction counts, RDS metrics, monitoring and slow query alerts.

   Remediation

   • Added an index to the interested-companies query on the course page
   • Changed how often the cache data stored in the database was reset and regenerated
   • Moved the read queries that loaded the writer DB to the reader DBs
   • Scaled the database to 8 r5.8xlarge instances, which bought only a few tens of minutes
   • Rewrote slow course queries or added indexes so they use indexes, and switched off features whose queries could not be fixed
   • Removed the large course description column from the select * course queries
   • Split the IDs of the IN clause into batches of 20 run with Promise.all
   • Upgraded PostgreSQL from 10.16 to 11.13
   • Set a 5-second query timeout
   • Raised max connections
   • Ran VACUUM by hand on the course table
   • Moved the large course description column into a separate table

   Action items

   • Change the structure in which one table's failure spreads to the whole service
   • Improve the table and cache structures that are hard to tune
   • Make the SQL code easier to follow
   • Improve code that gives no confidence to change, such as code without types or test code
   • Build a stable batch environment
   • Keep a monolith of reasonable size and separate the domains through which failures spread most

   Lessons

   Because every database bottleneck appeared as slow queries on the reader DBs, watching only the readers hid the root cause on the writer. In PostgreSQL, frequent updates to a table with large columns produce large dead tuples in bulk and slow every query on that table. Before the second event the team could not run a meaningful performance test because the first week showed that almost any page could fail under load. The event exposed high sensitivity to failure in every area, one table's failure spreading to the whole service, hard-to-tune table and cache structures, unintuitive SQL code, code without types or tests, no stable batch environment, and many hidden failure points not yet found.

   View case

By root cause4