Cases3 cases

1 to 3 of 3

1. IL-0432 AutoReserve notifications silently went unsent as Memorystore for Redis 3.2 rejected writes with OOM below its limit

   Hello Inc.2026

   Symptoms

   A bug report said that some reservation notifications in the AutoReserve restaurant reservation service never arrived. Of the several kinds of notification sent as a reservation progresses, only one kind sometimes arrived and sometimes did not.

   Writes rejectedMemory pressure

   Root cause

   Capacity exhaustion

   Component

   In-memory cache

   Contributing factor

   Monitoring blind spotUnsafe failure behaviourUnverified assumption

   Detection

   A bug report from AutoReserve said that some reservation notifications were not arriving.

   Remediation

   • Scaled the Memorystore for Redis instance from 8 GiB to 16 GiB, which did not stop the write rejections
   • Upgraded Redis from 3.2 to 7.2, which removed the fragmentation and stopped the write rejections
   • Moved the notification sent status from Redis to the relational database

   Action items

   • Consider enabling active defrag if physical memory use rises above 1.3 times logical memory use

   Lessons

   The logs and the instance page metrics did not show the cause; it took looking inside Redis with INFO memory to see physical memory far above logical memory. A cache is a place for values that can be rebuilt if lost and whose writes may fail, so the sent status, which had to persist and be decided once under concurrent processing, belonged in the relational database.

   View case
2. IL-0299 About 60 indices of Wikimedia's CirrusSearch OpenSearch clusters went red and unwritable during a host decommission

   Wikimedia Foundation2025

   Symptoms

   From 21:08 on 28 May 2025, about 60 indices of Wikimedia's OpenSearch clusters for CirrusSearch were in red, unwritable state. At 21:27 CirrusStreamingUpdaterFlinkJobUnstable alerts began to fire because the search update pipeline could not write to those indices.

   Writes rejected

   Trigger

   A Puppet patch that removed cluster access for the hosts elastic/cirrussearch10[53-67] while some of them still held primary shards.

   Root cause

   Configuration change

   Component

   Elasticsearch or OpenSearch

   Contributing factor

   Unverified assumptionNo runbookUntested backup

   Detection

   CirrusStreamingUpdaterFlinkJobUnstable alerts fired from 21:27; the review records the detection as human, by the engineer who started troubleshooting the next morning, and many other alerts were suppressed for maintenance.

   Remediation

   • Deleted the red indices, which brought the cluster back to green
   • Restored the broken indices

   Action items

   • Re-enable the snapshot repositories in production CirrusSearch
   • Create an operations-focused OpenSearch dashboard
   • Verify or fix the Logstash pipeline for Search Platform-owned OpenSearch clusters

   Lessons

   Banning nodes before removing them is a best-effort action and did not work as expected. A large operation such as decommissioning many servers at once had no maintenance plan or runbook, and the snapshots, dashboards and logging that recovery needed were not available or not documented well enough.

   View case
3. IL-0241 A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one day

   Stripe2019

   Symptoms

   On 10 July 2019 the Stripe API failed a substantial majority of requests twice, from 16:35 to 17:02 UTC and from 21:14 to 22:47 UTC. In the first period automated monitoring detected within a minute that a database cluster had failed to elect a new primary.

   High error rateFailover failureWrites rejectedIntermittent timeoutsSustained high node CPU

   Trigger

   The primary node of a database cluster failed at 16:35 UTC while one shard had several stalled nodes. The second period was set off by rolling the election protocol back to the previous minor version at 20:42 UTC.

   Root cause

   Software defect

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Change with a delayed effectMonitoring blind spotExcessive blast radiusHarmful manual actionUnverified assumptionOverlapping failures

   Detection

   Automated monitoring detected the failed election within a minute, and the team was alerted at 16:36 UTC. The second period was announced at 21:14 UTC by automated alerts that some shards in the cluster were unavailable.

   Remediation

   • Restarted all nodes of the database cluster, which forced a successful primary election
   • Updated the production configuration and restarted the database cluster's nodes
   • Ramped traffic back up, putting services for user-initiated API requests first

   Action items

   • Alert when nodes stop reporting their replication lag
   • Alert when a shard enters a state that could trigger the election fault
   • Work with the database maintainers on a fix for the underlying fault
   • Add circuit breaking on failed operations to particular database clusters
   • Use fault isolation to contain the impact of a single failed shard
   • Limit the resources used by clients that keep retrying failed requests
   • Add procedures and tooling that make rapid configuration changes during incident response safer

   View case

By root cause3