IL-0432 Hello Inc. · 2026-10-27 (published)databasegcp

AutoReserve notifications silently went unsent as Memorystore for Redis 3.2 rejected writes with OOM below its limit

Respo and AutoReserve are one Ruby on Rails application that share the relational database and Redis, run on Google Cloud with Memorystore for Redis as the cache. Because one notification can be sent through several paths, the application recorded a sent status in Redis with Rails.cache.write using unless_exist: true and a 3-minute expiry, and skipped sending when the write did not succeed. Rails.cache.write returns false when the key already exists and nil when the write itself fails, and Rails does not raise an exception on a failed cache write. The code did not handle nil, so every failed write was treated as already sent and logged "Notification already scheduled". The author had assumed an exception would be raised, which slowed the investigation.

Two checks decide whether memory is short on Memorystore. Redis looks at its logical memory use, the total of the values it holds, and when that reaches its limit it evicts keys with an expiry and only then rejects writes. Memorystore looks at the physical memory of the Redis process plus the OS and management processes, and rejects writes regardless of Redis's own limit or eviction. The physical figure is not on the instance page and is visible with INFO memory. The 6.71 GiB on the instance page was logical use; physical use was 8.81 GiB, 1.31 times the logical figure. Redis saw free room and evicted nothing, while the instance's physical memory moved back and forth across the threshold, so writes were rejected now and then. The gap was memory fragmentation: jemalloc can return memory to the OS only in whole pages, freed space is reused only by values of the same size, and cache values of varying sizes kept adding pages. Google Cloud's documentation states that fragmentation can leave a Memorystore instance out of memory even when the used-memory ratio is low.

The team first scaled the instance from 8 GiB to 16 GiB, but writes were still rejected, because the cached data grew with the limit and the fragmentation grew with it. The rejections stopped when Redis 3.2 was upgraded to 7.2: logical use went from 12.45 GiB to 10.81 GiB, physical use from 17.45 GiB to 10.94 GiB, and the ratio from 1.40 to 1.01, because the new process laid the data out again. In about four months after the upgrade no write rejection was recorded, including periods when logical use reached the limit and keys were evicted, and the ratio stayed between 1.0 and 1.2. The team guesses that faster removal of expired keys in Redis 6.0 changed how free space appears, but has not confirmed what made the difference. The sent status was also moved from Redis to the relational database, using an UPDATE that sets notified_at only on a row where it is empty, so the one process whose update succeeds sends the notification.

Severity

–

Scope

One kind of AutoReserve reservation notification was intermittently not sent to users

Users affected

–

Region

–

Time to detect

–

Time to resolve

–