Cases8 cases

1 to 5 of 8

1. IL-0432 AutoReserve notifications silently went unsent as Memorystore for Redis 3.2 rejected writes with OOM below its limit

   Hello Inc.2026

   Symptoms

   A bug report said that some reservation notifications in the AutoReserve restaurant reservation service never arrived. Of the several kinds of notification sent as a reservation progresses, only one kind sometimes arrived and sometimes did not.

   Writes rejectedMemory pressure

   Root cause

   Capacity exhaustion

   Component

   In-memory cache

   Contributing factor

   Monitoring blind spotUnsafe failure behaviourUnverified assumption

   Detection

   A bug report from AutoReserve said that some reservation notifications were not arriving.

   Remediation

   • Scaled the Memorystore for Redis instance from 8 GiB to 16 GiB, which did not stop the write rejections
   • Upgraded Redis from 3.2 to 7.2, which removed the fragmentation and stopped the write rejections
   • Moved the notification sent status from Redis to the relational database

   Action items

   • Consider enabling active defrag if physical memory use rises above 1.3 times logical memory use

   Lessons

   The logs and the instance page metrics did not show the cause; it took looking inside Redis with INFO memory to see physical memory far above logical memory. A cache is a place for values that can be rebuilt if lost and whose writes may fail, so the sent status, which had to persist and be decided once under concurrent processing, belonged in the relational database.

   View case
2. IL-0326 A retention clean-up job filled the disk of MailPace's managed Postgres, which stayed offline for just over two hours

   MailPace2023

   Symptoms

   At 13:36 BST on Friday August 11th 2023, minutes after MailPace reran its data clean-up job with more memory, notifications arrived that all database transactions were failing across multiple services. The managed Postgres database had gone offline, did not come back when restarted, and reported that it was out of space.

   Node disk fullService outageWrites rejected

   Trigger

   The daily retention clean-up job, which had been failing with killed, was rerun with its memory raised from 1gb to 8gb, and its large destroy_all deletions filled the database disk.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   Inefficient queryMonitoring blind spotNo headroom

   Detection

   Notifications that all database transactions were failing across multiple services arrived as soon as the outage began; the provider was automatically notified at 99% storage. MailPace's two status pages did not detect the outage.

   Remediation

   • Contacted the managed database provider's support
   • Notified users on Twitter and Mastodon
   • Prepared an offsite backup from six hours earlier on a new database instance as a fallback
   • The provider migrated the database to a larger instance
   • Removed the old data in production by hand, day by day, with an updated query

   Action items

   • Delete old data in batches of 50 records with in_batches in the clean-up job
   • Run the clean-up job four times a day instead of once
   • Stay on a much larger database instance with room to grow
   • Add cron monitoring to all cron jobs
   • Point the status pages at a /health endpoint that checks database access and returns a 500 error on failure
   • Investigate a hot-replica database for automated failover

   Lessons

   Several causes came together: a clean-up job that failed silently, a naive deletion that grew huge once data had piled up, a database plan without headroom for growth, and status pages that missed the outage, presumably because https://app.mailpace.com still answered 200 while the database was down. MailPace also advises that critical services such as email sending be set up with more than one provider.

   View case
3. IL-0302 Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022

   GitHub2022

   Symptoms

   On March 16, 2022 at 14:09 UTC, write operations across GitHub stopped working, including git operations, webhooks, pull requests, API requests, issues, GitHub Packages, GitHub Codespaces, GitHub Actions and GitHub Pages. Under peak-hour load, the database proxy in front of the shared mysql1 MySQL database had reached its maximum number of connections.

   Writes rejectedDatabase overloadHigh error rateFailover failure

   Trigger

   Increased load on mysql1 during peak hours, combined with poor query performance in specific circumstances, pushed the database proxy to its maximum number of connections.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Inefficient queryChange during peak hoursHarmful manual actionExcessive blast radius

   Remediation

   • Failed over mysql1 to a healthy replica on March 16
   • Reset the application connections to the new primary on March 17
   • Added an index that fixed the main query performance problem
   • Performed a primary failover on March 22
   • Performed a primary failover on March 23
   • Throttled webhook traffic to reduce load

   Action items

   • Keep throttling webhook traffic at peak load times
   • Audit the load patterns of mysql1 during peak hours and make performance fixes based on the audits
   • Move traffic to other databases to reduce load and speed up failover
   • Review change management, particularly monitoring and changes during high load in production
   • Shard the databases and scale up hardware

   View case
4. IL-0211 A Mandrill PostgreSQL shard shut down at transaction ID wraparound and email sending was impaired for about 40 hours

   Mailchimp2019

   Symptoms

   From 05:35 UTC on 4 February 2019 Mandrill, Mailchimp's transactional email service, sent only 80% of queued emails, because one of its PostgreSQL shards, shard4, stopped accepting writes. When Mandrill engineers opened an investigation at 11:58 UTC, they saw a sharply elevated load on shard4.

   Writes rejectedThroughput degradationLoad imbalanceNode disk full

   Trigger

   At 05:35 UTC on 4 February the transaction ID counter of shard4, the busiest Mandrill PostgreSQL shard, reached its upper limit and PostgreSQL shut the database down to protect against wraparound.

   Root cause

   Capacity exhaustion

   Component

   DatabasePostgreSQL vacuum

   Contributing factor

   Ignored early signalNo alert on limitSignal buried in noiseRetry stormMonitoring blind spot

   Detection

   The first exception was logged overnight but was buried by unrelated exceptions in the centralized error reporting. Mandrill engineers opened an investigation at 11:58 UTC and, while looking for the source of the rising load on shard4, found the wraparound at 12:56 UTC.

   Remediation

   • Put shard4 into standalone mode and started VACUUM at 14:16 UTC on 4 February
   • Stopped and restarted the vacuum with settings that made it run faster
   • Started a dump and restore of the database without the Search and Url tables while the vacuum ran
   • Replaced storage volumes on the Mandrill app servers whose disks were filling with queued jobs and logs
   • Moved the job-queue tables from shard4 to other live shards
   • Truncated the Search and Url tables on the locked database at 19:12 UTC on 5 February, after which the vacuum finished in about an hour
   • Started sending from the backlog at 21:36 UTC
   • Added alerting for XID wraparound by 22:40 UTC

   Lessons

   Mailchimp named the factors that slowed its response: knowledge of and access to Mandrill systems and monitoring rested with a few people, Mandrill logging needed a locally writable disk, logs of malformed API requests were hard to separate from real application errors, job health was hard to see, and the disk space problem made partial deployments for triage risky. What helped was trained incident commanders with clear roles (incident commanders in 3-hour rotations, scribes, technical liaisons and points of contact between teams), periodic summaries with a list of recent and open decisions, and taking time in a video meeting to reassess, which led to the truncation. Running in the cloud let the team expand storage quickly, and the team was lucky that the largest table holding up the vacuum held data it could delete.

   View case
5. IL-0300 Basecamp 3 went read-only for almost five hours when its events table reached the ceiling of 2,147,483,647

   37signals2018

   Symptoms

   On November 9th, 2018, from 7:21am to 12:11pm CST, Basecamp 3 was in read-only mode. Users could open existing messages, to-do lists and files, but could not add anything new or change anything that was already there.

   Writes rejected

   Trigger

   The busy events table, which records almost every activity in Basecamp 3, reached its ceiling of 2,147,483,647, and new events could not be written.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No capacity planning

   Remediation

   • Raised the ceiling of the tables from 2,147,483,647 to 9,223,372,036,854,775,807, the same change as the Rails 5.1 default

   Lessons

   The outage was avoidable. The team did not account for how quickly the events table was running out of headroom while it was already working on the problem, and it should have reviewed its own database schema when Rails 5.1 announced the new default in 2017.

   View case

By root cause1