Cases2 cases

1 to 2 of 2

1. IL-0311 BattleMetrics' primary PostgreSQL went read-only at transaction ID wraparound, breaking the website and RCON updates

   BattleMetrics2022

   Symptoms

   At 03:23 UTC on 27 March 2022, BattleMetrics' primary PostgreSQL database, which holds all account, server and player information, went into read-only mode. Users first saw the website fail to load, and soon RCON updates and queries began to fail as well.

   Writes rejectedService outage

   Trigger

   A corruption of the index on the player identifiers table, which BattleMetrics' logs place at about 26 February 2022, made every vacuum job fail, so transaction IDs kept climbing until Postgres stopped accepting writes on 27 March.

   Root cause

   Data loss or corruption

   Component

   DatabasePostgreSQL vacuumDatabase replicationDatabase backups and snapshots

   Contributing factor

   Monitoring blind spotOverlapping failuresNo headroom

   Detection

   Users failed to load the website and reported it as their API keys expired, then RCON updates and queries began to fail, and the team found that the main database no longer accepted writes.

   Remediation

   • Shut the database down and dropped the corrupted player identifiers index in single-user mode
   • Ran the vacuum again, which succeeded and let the database accept writes
   • Rebuilt the player identifier index, which took about two hours
   • Shut down the query workers and gave RCON updates priority
   • Pointed services at the new Postgres primary
   • Bootstrapped new replicas one after another to take load off the primary
   • Restarted queries with throttling, prioritised by server population and rank, after RCON updates had caught up

   Action items

   • Monitor errors in the Postgres logs
   • Monitor and alert on transaction IDs
   • Set up a logical replica in addition to physical replication
   • Build a faster network to speed up recovery from failure

   Lessons

   BattleMetrics names skipping health checks on transaction ID size as its critical mistake: the index corruption silently broke vacuum for a month, and without monitoring of transaction IDs nobody saw the database approach wraparound. Because the corruption had been replicated to the replicas, failing over did not help, and the backup servers and downloadable backups carried the same corruption. The team also found that, contrary to the error message, a full vacuum on a server in wraparound protection does not need single-user mode, and chose to run a slow, delayed service over no service while replicas were rebuilt.

   View case
2. IL-0486 Figma's PostgreSQL 9 database on RDS slowed under high write IOPS and lock contention until the site was unavailable

   Figma2020

   Symptoms

   On January 22, 2020 at 9:45 AM PST, Figma's database metrics showed rising write IOPS and lock contention, while database CPU was lower than normal. An automated alert followed at 10:00 AM, and by 11:42 AM enough API requests had queued up behind the database that Figma became unavailable for some users.

   Lock contention or deadlockDatabase overloadService outage

   Trigger

   An automatic ANALYZE on one of the tables in the query finished at exactly the moment the incident began. The new statistics pushed the planner's row estimate for an inner subquery past 200 rows, and the plan for the whole query switched to an estimate of over 20 million rows.

   Root cause

   Software defect

   Component

   DatabasePostgreSQL vacuum

   Contributing factor

   Overlapping failuresHarmful manual action

   Detection

   Database metrics showed higher write IOPS and lock contention at 9:45 AM PST, and an automated alert at 10:00 AM started the investigation.

   Remediation

   • Cancelled several inessential queries and increased the allocated IOPS
   • Cancelled autovacuum statements running on the largest tables
   • Changed autovacuum_freeze_max_age and restarted the database to stop aggressive autovacuuming
   • Took the service down for emergency maintenance and upgraded the database to PostgreSQL 11

   Action items

   • Revisit transaction timeouts and limits so long-running and expensive queries are terminated
   • Improve metrics, monitoring and database settings for autovacuuming, locking and query performance
   • Audit queries for unnecessary or excessive joins and subqueries that could trip up the query planner
   • Bring the performance tools of the upgraded PostgreSQL into the incident response plans
   • Audit the structure and access patterns of data stored in PostgreSQL and consider moving some of it to datastores suited to those patterns
   • Improve in-app messaging about unexpected maintenance windows
   • Move towards online database upgrades that do not need downtime
   • Let users keep working on already open files safely while offline, even after closing the browser

   Lessons

   Having upgraded staging to PostgreSQL 11, tested it for months and written a production upgrade plan let the team upgrade under poor conditions with confidence. A routine statistics refresh can flip a query to a catastrophic plan, and the same query's plan can differ widely over time, so comparing plans from snapshots taken at different dates exposed the cause.

   View case

By root cause2