Cases13 cases

11 to 13 of 13

1. IL-0312 Gadget's main Postgres did not start again after a restart to raise max_connections, and the platform went down

   Gadget2024

   Symptoms

   At 11:25 AM EST on 13 December 2024, Gadget restarted its main PostgreSQL server in place to raise max_connections. Postgres shut down but did not start again, and every request to Gadget applications failed, including frontend requests, GraphQL API requests, webhook processing and syncs.

   Service outageHigh latency

   Trigger

   An emergency in-place restart of the main Postgres server at 11:25 AM EST, decided at 11:00 AM, to raise max_connections, which Postgres cannot change without a full restart.

   Root cause

   Capacity exhaustion

   Component

   DatabaseLinux kernelDatabase proxy and connection pooler

   Contributing factor

   Unverified assumptionUnrepresentative stagingAutomation amplificationExcessive blast radius

   Detection

   The team carrying out the planned restart saw at 11:26 AM EST that Postgres had shut down and did not start again.

   Remediation

   • Restarted the database server with a hardware reset, which cleared the fragmented memory so Postgres could start
   • Provisioned almost 4x the compute after the autoscalers had scaled it down during the outage

   Action items

   • Use only the replica-failover approach for disruptive database operations such as restarts
   • Run a more aggressive capacity planning process before Black Friday / Cyber Monday, preparing for a 6x increase in load and proving it with load tests well in advance
   • Announce planned downtime ahead of time on the status page and community Discord, and make the operating procedures tell planned maintenance apart from service disruptions
   • Finish the database sharding project so apps can use different databases and configuration changes can go through canary and blue/green setups
   • Finish building a zero-downtime database failover system
   • Set much higher scaling floors on the compute autoscalers

   Lessons

   Gadget names three deeper factors. Choosing the in-place restart put the database into a state the team had not seen before and was not confident in. Capacity planning for Black Friday / Cyber Monday assumed a doubling of traffic instead of the almost 5x that came, so settings such as max_connections stopped fitting and had to be changed in an emergency instead of in a maintenance window practised in staging. And there was no way to test database configuration changes in a production-like environment: integration tests run against a synthetic Postgres database that never goes through the months of production load behind the memory fragmentation or the memory usage issues.

   View case
2. IL-0308 Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slow

   RevenueCat2022

   Symptoms

   At 9:34 UTC on November 23, 2022, RevenueCat started moving traffic from its Aurora PostgreSQL 10.x cluster to a new Aurora PostgreSQL 14.x cluster. By 9:39 UTC all traffic was on the new cluster, latency was bad, and the RevenueCat dashboard and the endpoints used by the RevenueCat SDK stopped working.

   Database overloadHigh latencyService outageConnection pool exhaustion

   Trigger

   At 9:34 UTC production traffic was switched from the Aurora PostgreSQL 10.x cluster to the new Aurora PostgreSQL 14.x cluster, before ANALYZE had run on its largest tables.

   Root cause

   Breaking change in an upgrade

   Component

   DatabasePostgreSQL vacuumDatabase replicationDatabase proxy and connection poolerQuery planner and table statistics

   Contributing factor

   Unrepresentative stagingUnverified assumptionAutomation amplificationMonitoring blind spot

   Detection

   At 9:39 UTC, with all traffic on the new cluster, latency looked bad and the infrastructure team began to investigate.

   Remediation

   • Added a statement timeout to the database user so queries could not hold a connection too long
   • Shed 90% of traffic by returning 503s from the load balancer
   • Dropped indexes that the planner was using wrongly and that were not needed
   • Stopped accepting Apple webhooks for a time
   • Rolled out a release without New Relic to stop its EXPLAIN queries
   • Ran ANALYZE by hand on the biggest tables
   • Raised the values of the affected primary key sequences with a prepared script
   • Stopped the integration event dispatcher by hand
   • Replayed the webhooks that were dispatched but not recorded
   • Restored integration event dispatching, ETLs and other non-web server jobs

   Action items

   • Consider change freezes during business-critical periods
   • Explain better how the SDK behaves during backend outages
   • Tell customers about maintenance and outages in the developer dashboard and by direct email to technical contacts
   • Replay a subset of write traffic in migration dry runs to mimic the cutover
   • Move costly processes to asynchronous work to keep the API fast and stable
   • Re-evaluate table indexes, including unused ones the database may choose for queries
   • Show the load of each endpoint and system in the database dashboards
   • Involve support and customer communications staff during outages

   Lessons

   The new cluster had not been analyzed, and the tests missed it because the test cluster had been up long enough to gather statistics. Dry runs replayed no write traffic, on the belief that writes are simpler than reads, and the team assumed sequence state was replicated with the data. Rolling back at the start would have been better than fixing forward. The migration, timed to finish before Thanksgiving and Black Friday, should have been delayed until the week after, because the US had entered a sensitive period and part of the US support team was already on holiday.

   View case
3. IL-0302 Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022

   GitHub2022

   Symptoms

   On March 16, 2022 at 14:09 UTC, write operations across GitHub stopped working, including git operations, webhooks, pull requests, API requests, issues, GitHub Packages, GitHub Codespaces, GitHub Actions and GitHub Pages. Under peak-hour load, the database proxy in front of the shared mysql1 MySQL database had reached its maximum number of connections.

   Writes rejectedDatabase overloadHigh error rateFailover failure

   Trigger

   Increased load on mysql1 during peak hours, combined with poor query performance in specific circumstances, pushed the database proxy to its maximum number of connections.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Inefficient queryChange during peak hoursHarmful manual actionExcessive blast radius

   Remediation

   • Failed over mysql1 to a healthy replica on March 16
   • Reset the application connections to the new primary on March 17
   • Added an index that fixed the main query performance problem
   • Performed a primary failover on March 22
   • Performed a primary failover on March 23
   • Throttled webhook traffic to reduce load

   Action items

   • Keep throttling webhook traffic at peak load times
   • Audit the load patterns of mysql1 during peak hours and make performance fixes based on the audits
   • Move traffic to other databases to reduce load and speed up failover
   • Review change management, particularly monitoring and changes during high load in production
   • Shard the databases and scale up hardware

   View case

By root cause6