Cases2 cases

1 to 2 of 2

1. IL-0432 AutoReserve notifications silently went unsent as Memorystore for Redis 3.2 rejected writes with OOM below its limit

   Hello Inc.2026

   Symptoms

   A bug report said that some reservation notifications in the AutoReserve restaurant reservation service never arrived. Of the several kinds of notification sent as a reservation progresses, only one kind sometimes arrived and sometimes did not.

   Writes rejectedMemory pressure

   Root cause

   Capacity exhaustion

   Component

   In-memory cache

   Contributing factor

   Monitoring blind spotUnsafe failure behaviourUnverified assumption

   Detection

   A bug report from AutoReserve said that some reservation notifications were not arriving.

   Remediation

   • Scaled the Memorystore for Redis instance from 8 GiB to 16 GiB, which did not stop the write rejections
   • Upgraded Redis from 3.2 to 7.2, which removed the fragmentation and stopped the write rejections
   • Moved the notification sent status from Redis to the relational database

   Action items

   • Consider enabling active defrag if physical memory use rises above 1.3 times logical memory use

   Lessons

   The logs and the instance page metrics did not show the cause; it took looking inside Redis with INFO memory to see physical memory far above logical memory. A cache is a place for values that can be rebuilt if lost and whose writes may fail, so the sent status, which had to persist and be decided once under concurrent processing, belonged in the relational database.

   View case
2. IL-0296 Transloadit's us-east Redis cluster lost queued jobs and heartbeats, raising WORKER_JOB_ERROR errors for 81 customers

   Transloadit2016

   Symptoms

   Reports came in of more WORKER_JOB_ERROR errors, which tell Transloadit there is a communication problem between its Uploaders, the machines that receive files and orchestrate Assemblies, and its Drones, the machines that run encoding tasks. Many Assemblies became much slower because retries with exponential back-off kicked in, and a few could not be replayed.

   High error rateHigh latencyLoad imbalance

   Trigger

   A steep increase in HLS traffic, together with more Drones online processing the same volume faster, which put more load on the central Redis nodes.

   Root cause

   Capacity exhaustion

   Component

   In-memory cacheDatabase replication

   Contributing factor

   Monitoring blind spotUnverified assumptionHarmful manual action

   Detection

   Reports of an increased number of WORKER_JOB_ERROR errors.

   Remediation

   • Re-added lost messages and relaxed heartbeat monitoring as a stopgap
   • Upgraded Redis to the latest stable version
   • Doubled the Redis cluster to six nodes and then added three more, which made it worse
   • Brought the number of read replicas down to one, which cut traffic and failure rates
   • Capped how far Drones scale up in each region

   Action items

   • Add logging and alarms for approaching saturation
   • Find ways to decentralise the traffic between the Redis nodes and the Drones and Uploaders
   • Add sharding per Uploader on top of the sharding per region

   Lessons

   To handle more traffic Transloadit had to use fewer Redis nodes, not more: each added replica multiplied the replication traffic from the single leader that takes every write, so scaling up the number of nodes was the wrong call. Bandwidth had not been among the monitored Redis metrics.

   View case

By root cause1