Incident Library
Applied filters ComponentQuery planner and table statisticsSymptomConnection pool exhaustionOrganisationRevenueCatClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0308 Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slow
    RevenueCat2022
    Symptoms

    At 9:34 UTC on November 23, 2022, RevenueCat started moving traffic from its Aurora PostgreSQL 10.x cluster to a new Aurora PostgreSQL 14.x cluster. By 9:39 UTC all traffic was on the new cluster, latency was bad, and the RevenueCat dashboard and the endpoints used by the RevenueCat SDK stopped working.

    Database overloadHigh latencyService outageConnection pool exhaustion
    Trigger

    At 9:34 UTC production traffic was switched from the Aurora PostgreSQL 10.x cluster to the new Aurora PostgreSQL 14.x cluster, before ANALYZE had run on its largest tables.

    Root cause
    Breaking change in an upgrade
    Component
    DatabasePostgreSQL vacuumDatabase replicationDatabase proxy and connection poolerQuery planner and table statistics
    Contributing factor
    Unrepresentative stagingUnverified assumptionAutomation amplificationMonitoring blind spot
    Detection

    At 9:39 UTC, with all traffic on the new cluster, latency looked bad and the infrastructure team began to investigate.

    Remediation
    • Added a statement timeout to the database user so queries could not hold a connection too long
    • Shed 90% of traffic by returning 503s from the load balancer
    • Dropped indexes that the planner was using wrongly and that were not needed
    • Stopped accepting Apple webhooks for a time
    • Rolled out a release without New Relic to stop its EXPLAIN queries
    • Ran ANALYZE by hand on the biggest tables
    • Raised the values of the affected primary key sequences with a prepared script
    • Stopped the integration event dispatcher by hand
    • Replayed the webhooks that were dispatched but not recorded
    • Restored integration event dispatching, ETLs and other non-web server jobs
    Action items
    • Consider change freezes during business-critical periods
    • Explain better how the SDK behaves during backend outages
    • Tell customers about maintenance and outages in the developer dashboard and by direct email to technical contacts
    • Replay a subset of write traffic in migration dry runs to mimic the cutover
    • Move costly processes to asynchronous work to keep the API fast and stable
    • Re-evaluate table indexes, including unused ones the database may choose for queries
    • Show the load of each endpoint and system in the database dashboards
    • Involve support and customer communications staff during outages
    Lessons

    The new cluster had not been analyzed, and the tests missed it because the test cluster had been up long enough to gather statistics. Dry runs replayed no write traffic, on the belief that writes are simpler than reads, and the team assumed sequence state was replicated with the data. Rolling back at the start would have been better than fixing forward. The migration, timed to finish before Thanksgiving and Black Friday, should have been delayed until the week after, because the US had entered a sensitive period and part of the US support team was already on holiday.

    View case
Check your cluster for this

This case involves PostgreSQL vacuum and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade1 case

Component
PostgreSQL vacuumDatabaseDatabase proxy and connection poolerDatabase replicationQuery planner and table statistics
Contributing factor
Unverified assumptionAutomation amplificationMonitoring blind spotUnrepresentative staging
Organisation
RevenueCat
  1. IL-0308 Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slow
    RevenueCat2022