Cases8 cases

5 to 8 of 8

1. IL-0308 Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slow

   RevenueCat2022

   Symptoms

   At 9:34 UTC on November 23, 2022, RevenueCat started moving traffic from its Aurora PostgreSQL 10.x cluster to a new Aurora PostgreSQL 14.x cluster. By 9:39 UTC all traffic was on the new cluster, latency was bad, and the RevenueCat dashboard and the endpoints used by the RevenueCat SDK stopped working.

   Database overloadHigh latencyService outageConnection pool exhaustion

   Trigger

   At 9:34 UTC production traffic was switched from the Aurora PostgreSQL 10.x cluster to the new Aurora PostgreSQL 14.x cluster, before ANALYZE had run on its largest tables.

   Root cause

   Breaking change in an upgrade

   Component

   DatabasePostgreSQL vacuumDatabase replicationDatabase proxy and connection poolerQuery planner and table statistics

   Contributing factor

   Unrepresentative stagingUnverified assumptionAutomation amplificationMonitoring blind spot

   Detection

   At 9:39 UTC, with all traffic on the new cluster, latency looked bad and the infrastructure team began to investigate.

   Remediation

   • Added a statement timeout to the database user so queries could not hold a connection too long
   • Shed 90% of traffic by returning 503s from the load balancer
   • Dropped indexes that the planner was using wrongly and that were not needed
   • Stopped accepting Apple webhooks for a time
   • Rolled out a release without New Relic to stop its EXPLAIN queries
   • Ran ANALYZE by hand on the biggest tables
   • Raised the values of the affected primary key sequences with a prepared script
   • Stopped the integration event dispatcher by hand
   • Replayed the webhooks that were dispatched but not recorded
   • Restored integration event dispatching, ETLs and other non-web server jobs

   Action items

   • Consider change freezes during business-critical periods
   • Explain better how the SDK behaves during backend outages
   • Tell customers about maintenance and outages in the developer dashboard and by direct email to technical contacts
   • Replay a subset of write traffic in migration dry runs to mimic the cutover
   • Move costly processes to asynchronous work to keep the API fast and stable
   • Re-evaluate table indexes, including unused ones the database may choose for queries
   • Show the load of each endpoint and system in the database dashboards
   • Involve support and customer communications staff during outages

   Lessons

   The new cluster had not been analyzed, and the tests missed it because the test cluster had been up long enough to gather statistics. Dry runs replayed no write traffic, on the belief that writes are simpler than reads, and the team assumed sequence state was replicated with the data. Rolling back at the start would have been better than fixing forward. The migration, timed to finish before Thanksgiving and Black Friday, should have been delayed until the week after, because the US had entered a sensitive period and part of the US support team was already on holiday.

   View case
2. IL-0301 Basecamp 3's primary database went into total gridlock at the top of the hour and Basecamp 3 went down twice

   37signals2022

   Symptoms

   At 13:00 UTC on Monday, June 27, 2022, Basecamp 3 became extremely slow, then showed error pages, and then stopped working altogether. Basecamp 2 and Basecamp Classic kept working normally.

   Lock contention or deadlockDatabase overloadHigh latencyHigh error rateService outage

   Trigger

   At 13:00 UTC, when Basecamp 3 sends scheduled notifications in massive batches, a rush of demanding queries reached the primary database and started a cascade of heavy lock contention.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No capacity planningHarmful manual actionMonitoring blind spot

   Detection

   The team saw the Basecamp 3 primary database become completely congested within 30 seconds of 13:00 UTC and responded immediately.

   Remediation

   • Paused and disabled parts of Basecamp 3, such as delaying progress report emails
   • Failed over from the primary database to a replica, which brought Basecamp 3 back until it tipped over again
   • Put Basecamp 3 into maintenance mode
   • Set hard limits on database query time
   • Brought features back online step by step with the query limits in place

   Action items

   • Keep hard limits on database query time so slow queries fail instead of piling up
   • Tune the limits on concurrency
   • Treat top-of-the-hour database load spikes from scheduled batches as unacceptable
   • Reduce the number of parallel workers for some scheduled jobs
   • Let some scheduled activities run over several minutes instead of all at once
   • Move bursty queries from the primary database to read-only replicas
   • Fix the quiet performance regressions found in a review of the past year of database metrics
   • Find and spread out other bursty activities that concentrate load on the database
   • Tune the database for more predictable behaviour under heavy load and high concurrency
   • Add observability into the database's internal locking and transaction states
   • Build tools to switch off high-load Basecamp 3 features selectively during an incident
   • Run operational fire drills for a wedged database
   • Evaluate upgrading the database software and hardware
   • Set service level objectives for the components behind Basecamp 3
   • Hold quarterly performance and load reviews to catch slow, subtle changes

   Lessons

   A system running just past its limits can flip from stable to unstable without any particular trigger, and steady growth plus small regressions that raised no alarm had brought Basecamp 3's primary database to that point. Under lock contention, terminating queries does not relieve the database, because rolling back their transactions waits on the same locks. The team struggled to respond with confidence because there was no single clear cause and little visibility into the database's internals.

   View case
3. IL-0456 Inflearn went down six times during a January 2022 free-course event as slow queries piled up on Aurora PostgreSQL

   Inflab2022

   Symptoms

   During a 100% discount event that made 125 courses free, Inflearn received 5 to 20 times its normal traffic, and on four days, 3, 4, 7 and 17 January 2022, the service became unusable around 9 a.m. and 10 p.m. There were six outages in all.

   Service outageDatabase overloadHigh latency

   Trigger

   A 100% discount event on 125 courses, from 3 to 17 January and 18 to 24 January 2022, went viral and brought 5 to 20 times the usual traffic.

   Root cause

   Capacity exhaustion

   Component

   DatabasePostgreSQL vacuum

   Contributing factor

   Inefficient queryMonitoring blind spotExcessive blast radius

   Detection

   The team followed the outages through hourly transaction counts, RDS metrics, monitoring and slow query alerts.

   Remediation

   • Added an index to the interested-companies query on the course page
   • Changed how often the cache data stored in the database was reset and regenerated
   • Moved the read queries that loaded the writer DB to the reader DBs
   • Scaled the database to 8 r5.8xlarge instances, which bought only a few tens of minutes
   • Rewrote slow course queries or added indexes so they use indexes, and switched off features whose queries could not be fixed
   • Removed the large course description column from the select * course queries
   • Split the IDs of the IN clause into batches of 20 run with Promise.all
   • Upgraded PostgreSQL from 10.16 to 11.13
   • Set a 5-second query timeout
   • Raised max connections
   • Ran VACUUM by hand on the course table
   • Moved the large course description column into a separate table

   Action items

   • Change the structure in which one table's failure spreads to the whole service
   • Improve the table and cache structures that are hard to tune
   • Make the SQL code easier to follow
   • Improve code that gives no confidence to change, such as code without types or test code
   • Build a stable batch environment
   • Keep a monolith of reasonable size and separate the domains through which failures spread most

   Lessons

   Because every database bottleneck appeared as slow queries on the reader DBs, watching only the readers hid the root cause on the writer. In PostgreSQL, frequent updates to a table with large columns produce large dead tuples in bulk and slow every query on that table. Before the second event the team could not run a meaningful performance test because the first week showed that almost any page could fail under load. The event exposed high sensitivity to failure in every area, one table's failure spreading to the whole service, hard-to-tune table and cache structures, unintuitive SQL code, code without types or tests, no stable batch environment, and many hidden failure points not yet found.

   View case
4. IL-0282 DynamoDB in US-East reached about 55% request errors as storage servers overloaded its metadata service

   Amazon Web Services2015

   Symptoms

   Early on Sunday 20 September 2015, the error rate of customer requests to Amazon DynamoDB in the US-East Region rose far beyond any level of the previous 3 years by 2:37am PDT and settled at about 55%. Some other AWS services in the region that depend on DynamoDB were affected as well.

   High error rateDatabase overload

   Trigger

   At 2:19am PDT a brief network disruption affected a portion of DynamoDB's storage servers, and they all requested their membership data from the metadata service at the same time.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No capacity planningMonitoring blind spotRetry stormCircular dependencyExcessive blast radius

   Detection

   For the errors that remained after the main recovery, a growing number of customer support cases on Monday about tables stuck in the updating or deleting stage and higher than normal error rates showed that the low overall error rate was hitting some customers hard.

   Remediation

   • Paused requests to the metadata service, which cut the retries and relieved its load
   • Added significant capacity to the metadata service once it could answer administrative requests
   • Turned requests to the metadata service back on and returned storage servers to the customer request path
   • Restored a metadata partition that was still not taking its full traffic, ending the remaining errors on Monday

   Action items

   • Significantly increased the capacity of the metadata service, which AWS had already done by the time of the summary
   • Add stricter monitoring on performance dimensions such as membership size, to plan capacity ahead
   • Reduce the rate at which storage nodes request membership data
   • Lengthen the time allowed to process membership queries
   • Segment DynamoDB into many metadata service instances, each serving only part of the storage server fleet
   • Change SQS metadata caching so send and receive keep working without long access to the metadata table
   • Partition work across the Auto Scaling servers to process a backlog of scaling jobs in parallel
   • Prune older scaling activities that have been superseded
   • Increase the capacity available to process scaling activities
   • Change CloudWatch caching of DynamoDB group membership data to refresh only the smallest set of metrics
   • Deliver CloudWatch metrics faster through write-through caching
   • Roll out the shorter timeout on the DynamoDB-dependent API call in Console login

   Lessons

   A low overall error rate can hide disproportionately high error rates for a few customers, and AWS should have posted to the dashboard sooner. A call that does not need to succeed, like the DynamoDB-dependent API call in Console login, should fail quickly instead of blocking on a very long timeout. Splitting the metadata service into many instances, each serving part of the fleet, is meant to contain the impact of software, capacity or infrastructure failures.

   View case

By root cause4