Cases16 cases

1 to 5 of 16

1. IL-0432 AutoReserve notifications silently went unsent as Memorystore for Redis 3.2 rejected writes with OOM below its limit

   Hello Inc.2026

   Symptoms

   A bug report said that some reservation notifications in the AutoReserve restaurant reservation service never arrived. Of the several kinds of notification sent as a reservation progresses, only one kind sometimes arrived and sometimes did not.

   Writes rejectedMemory pressure

   Root cause

   Capacity exhaustion

   Component

   In-memory cache

   Contributing factor

   Monitoring blind spotUnsafe failure behaviourUnverified assumption

   Detection

   A bug report from AutoReserve said that some reservation notifications were not arriving.

   Remediation

   • Scaled the Memorystore for Redis instance from 8 GiB to 16 GiB, which did not stop the write rejections
   • Upgraded Redis from 3.2 to 7.2, which removed the fragmentation and stopped the write rejections
   • Moved the notification sent status from Redis to the relational database

   Action items

   • Consider enabling active defrag if physical memory use rises above 1.3 times logical memory use

   Lessons

   The logs and the instance page metrics did not show the cause; it took looking inside Redis with INFO memory to see physical memory far above logical memory. A cache is a place for values that can be rebuilt if lost and whose writes may fail, so the sent status, which had to persist and be decided once under concurrent processing, belonged in the relational database.

   View case
2. IL-0310 An Aurora PostgreSQL failover to an upsized reader reversed twice and backend writes failed in read-only transactions

   Hightouch2025

   Symptoms

   At 16:39 EDT on 23 October 2025, Hightouch triggered a failover of its Aurora PostgreSQL cluster in us-east-1 to a newly upsized reader. The AWS Console showed the usual parameter adjustments and instance restarts, but after the page refreshed the original writer was still the primary, and backend services could not run write queries although AWS showed the cluster as healthy.

   Failover failureWrites rejected

   Trigger

   A planned upsize of the Aurora cluster's instances, done without a maintenance window, after the AWS us-east-1 outage of 20 October had left an event backlog that strained the small Postgres instance. The manual failover to the upsized reader at 16:39 EDT started the failure.

   Root cause

   Software defect

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Unrepresentative staging

   Detection

   The AWS Console showed after a refresh that the original writer was still primary, and backend application logs showed write errors on a read-only transaction. Per-instance query monitoring in Datadog and the RDS database logs showed the brief writer advertisement.

   Remediation

   • Restarted the backend services, which cleared the write errors after each failed failover
   • Scaled down every service that writes to the database
   • Triggered the failover again with no concurrent writes, which completed, and upsized the cluster in us-east-1

   Action items

   • Pause writers before any intentional failover, as written into the internal playbooks
   • Monitor for unexpected flips of the instance that advertises the writer role

   Lessons

   Hightouch's takeaways are that a migration can end in the desired state, the starting state or somewhere in between even with services one trusts, so traffic redirection and brief dependency outages should be planned for. The brief writer advertisement could only be found because queries to each instance were monitored in Datadog and the RDS database logs were available. Limiting how much any single component can affect the system helps both uptime and maintenance, and a staging load test, even of the exact procedure, did not reproduce the conditions that triggered the Aurora race condition.

   View case
3. IL-0299 About 60 indices of Wikimedia's CirrusSearch OpenSearch clusters went red and unwritable during a host decommission

   Wikimedia Foundation2025

   Symptoms

   From 21:08 on 28 May 2025, about 60 indices of Wikimedia's OpenSearch clusters for CirrusSearch were in red, unwritable state. At 21:27 CirrusStreamingUpdaterFlinkJobUnstable alerts began to fire because the search update pipeline could not write to those indices.

   Writes rejected

   Trigger

   A Puppet patch that removed cluster access for the hosts elastic/cirrussearch10[53-67] while some of them still held primary shards.

   Root cause

   Configuration change

   Component

   Elasticsearch or OpenSearch

   Contributing factor

   Unverified assumptionNo runbookUntested backup

   Detection

   CirrusStreamingUpdaterFlinkJobUnstable alerts fired from 21:27; the review records the detection as human, by the engineer who started troubleshooting the next morning, and many other alerts were suppressed for maintenance.

   Remediation

   • Deleted the red indices, which brought the cluster back to green
   • Restored the broken indices

   Action items

   • Re-enable the snapshot repositories in production CirrusSearch
   • Create an operations-focused OpenSearch dashboard
   • Verify or fix the Logstash pipeline for Search Platform-owned OpenSearch clusters

   Lessons

   Banning nodes before removing them is a best-effort action and did not work as expected. A large operation such as decommissioning many servers at once had no maintenance plan or runbook, and the snapshots, dashboards and logging that recovery needed were not available or not documented well enough.

   View case
4. IL-0326 A retention clean-up job filled the disk of MailPace's managed Postgres, which stayed offline for just over two hours

   MailPace2023

   Symptoms

   At 13:36 BST on Friday August 11th 2023, minutes after MailPace reran its data clean-up job with more memory, notifications arrived that all database transactions were failing across multiple services. The managed Postgres database had gone offline, did not come back when restarted, and reported that it was out of space.

   Node disk fullService outageWrites rejected

   Trigger

   The daily retention clean-up job, which had been failing with killed, was rerun with its memory raised from 1gb to 8gb, and its large destroy_all deletions filled the database disk.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   Inefficient queryMonitoring blind spotNo headroom

   Detection

   Notifications that all database transactions were failing across multiple services arrived as soon as the outage began; the provider was automatically notified at 99% storage. MailPace's two status pages did not detect the outage.

   Remediation

   • Contacted the managed database provider's support
   • Notified users on Twitter and Mastodon
   • Prepared an offsite backup from six hours earlier on a new database instance as a fallback
   • The provider migrated the database to a larger instance
   • Removed the old data in production by hand, day by day, with an updated query

   Action items

   • Delete old data in batches of 50 records with in_batches in the clean-up job
   • Run the clean-up job four times a day instead of once
   • Stay on a much larger database instance with room to grow
   • Add cron monitoring to all cron jobs
   • Point the status pages at a /health endpoint that checks database access and returns a 500 error on failure
   • Investigate a hot-replica database for automated failover

   Lessons

   Several causes came together: a clean-up job that failed silently, a naive deletion that grew huge once data had piled up, a database plan without headroom for growth, and status pages that missed the outage, presumably because https://app.mailpace.com still answered 200 while the database was down. MailPace also advises that critical services such as email sending be set up with more than one provider.

   View case
5. IL-0311 BattleMetrics' primary PostgreSQL went read-only at transaction ID wraparound, breaking the website and RCON updates

   BattleMetrics2022

   Symptoms

   At 03:23 UTC on 27 March 2022, BattleMetrics' primary PostgreSQL database, which holds all account, server and player information, went into read-only mode. Users first saw the website fail to load, and soon RCON updates and queries began to fail as well.

   Writes rejectedService outage

   Trigger

   A corruption of the index on the player identifiers table, which BattleMetrics' logs place at about 26 February 2022, made every vacuum job fail, so transaction IDs kept climbing until Postgres stopped accepting writes on 27 March.

   Root cause

   Data loss or corruption

   Component

   DatabasePostgreSQL vacuumDatabase replicationDatabase backups and snapshots

   Contributing factor

   Monitoring blind spotOverlapping failuresNo headroom

   Detection

   Users failed to load the website and reported it as their API keys expired, then RCON updates and queries began to fail, and the team found that the main database no longer accepted writes.

   Remediation

   • Shut the database down and dropped the corrupted player identifiers index in single-user mode
   • Ran the vacuum again, which succeeded and let the database accept writes
   • Rebuilt the player identifier index, which took about two hours
   • Shut down the query workers and gave RCON updates priority
   • Pointed services at the new Postgres primary
   • Bootstrapped new replicas one after another to take load off the primary
   • Restarted queries with throttling, prioritised by server population and rank, after RCON updates had caught up

   Action items

   • Monitor errors in the Postgres logs
   • Monitor and alert on transaction IDs
   • Set up a logical replica in addition to physical replication
   • Build a faster network to speed up recovery from failure

   Lessons

   BattleMetrics names skipping health checks on transaction ID size as its critical mistake: the index corruption silently broke vacuum for a month, and without monitoring of transaction IDs nobody saw the database approach wraparound. Because the corruption had been replicated to the replicas, failing over did not help, and the backup servers and downloadable backups carried the same corruption. The team also found that, contrary to the error message, a full vacuum on a server in wraparound protection does not need single-user mode, and chose to run a slow, delayed service over no service while replicas were rebuilt.

   View case

By root cause7