Cases2 cases

1 to 2 of 2

1. IL-0326 A retention clean-up job filled the disk of MailPace's managed Postgres, which stayed offline for just over two hours

   MailPace2023

   Symptoms

   At 13:36 BST on Friday August 11th 2023, minutes after MailPace reran its data clean-up job with more memory, notifications arrived that all database transactions were failing across multiple services. The managed Postgres database had gone offline, did not come back when restarted, and reported that it was out of space.

   Node disk fullService outageWrites rejected

   Trigger

   The daily retention clean-up job, which had been failing with killed, was rerun with its memory raised from 1gb to 8gb, and its large destroy_all deletions filled the database disk.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   Inefficient queryMonitoring blind spotNo headroom

   Detection

   Notifications that all database transactions were failing across multiple services arrived as soon as the outage began; the provider was automatically notified at 99% storage. MailPace's two status pages did not detect the outage.

   Remediation

   • Contacted the managed database provider's support
   • Notified users on Twitter and Mastodon
   • Prepared an offsite backup from six hours earlier on a new database instance as a fallback
   • The provider migrated the database to a larger instance
   • Removed the old data in production by hand, day by day, with an updated query

   Action items

   • Delete old data in batches of 50 records with in_batches in the clean-up job
   • Run the clean-up job four times a day instead of once
   • Stay on a much larger database instance with room to grow
   • Add cron monitoring to all cron jobs
   • Point the status pages at a /health endpoint that checks database access and returns a 500 error on failure
   • Investigate a hot-replica database for automated failover

   Lessons

   Several causes came together: a clean-up job that failed silently, a naive deletion that grew huge once data had piled up, a database plan without headroom for growth, and status pages that missed the outage, presumably because https://app.mailpace.com still answered 200 while the database was down. MailPace also advises that critical services such as email sending be set up with more than one provider.

   View case
2. IL-0302 Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022

   GitHub2022

   Symptoms

   On March 16, 2022 at 14:09 UTC, write operations across GitHub stopped working, including git operations, webhooks, pull requests, API requests, issues, GitHub Packages, GitHub Codespaces, GitHub Actions and GitHub Pages. Under peak-hour load, the database proxy in front of the shared mysql1 MySQL database had reached its maximum number of connections.

   Writes rejectedDatabase overloadHigh error rateFailover failure

   Trigger

   Increased load on mysql1 during peak hours, combined with poor query performance in specific circumstances, pushed the database proxy to its maximum number of connections.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Inefficient queryChange during peak hoursHarmful manual actionExcessive blast radius

   Remediation

   • Failed over mysql1 to a healthy replica on March 16
   • Reset the application connections to the new primary on March 17
   • Added an index that fixed the main query performance problem
   • Performed a primary failover on March 22
   • Performed a primary failover on March 23
   • Throttled webhook traffic to reduce load

   Action items

   • Keep throttling webhook traffic at peak load times
   • Audit the load patterns of mysql1 during peak hours and make performance fixes based on the audits
   • Move traffic to other databases to reduce load and speed up failover
   • Review change management, particularly monitoring and changes during high load in production
   • Shard the databases and scale up hardware

   View case

By root cause1