Cases6 cases

1 to 5 of 6

1. IL-0432 AutoReserve notifications silently went unsent as Memorystore for Redis 3.2 rejected writes with OOM below its limit

   Hello Inc.2026

   Symptoms

   A bug report said that some reservation notifications in the AutoReserve restaurant reservation service never arrived. Of the several kinds of notification sent as a reservation progresses, only one kind sometimes arrived and sometimes did not.

   Writes rejectedMemory pressure

   Root cause

   Capacity exhaustion

   Component

   In-memory cache

   Contributing factor

   Monitoring blind spotUnsafe failure behaviourUnverified assumption

   Detection

   A bug report from AutoReserve said that some reservation notifications were not arriving.

   Remediation

   • Scaled the Memorystore for Redis instance from 8 GiB to 16 GiB, which did not stop the write rejections
   • Upgraded Redis from 3.2 to 7.2, which removed the fragmentation and stopped the write rejections
   • Moved the notification sent status from Redis to the relational database

   Action items

   • Consider enabling active defrag if physical memory use rises above 1.3 times logical memory use

   Lessons

   The logs and the instance page metrics did not show the cause; it took looking inside Redis with INFO memory to see physical memory far above logical memory. A cache is a place for values that can be rebuilt if lost and whose writes may fail, so the sent status, which had to persist and be decided once under concurrent processing, belonged in the relational database.

   View case
2. IL-0575 Amazon OpenSearch cluster went red under high heap usage and the admin screen's customer list would not open

   Sonicmoov2026

   Symptoms

   During a regular meeting, Sonicmoov's customer success team reported that the customer list in the admin screen would not open, and customer inquiries arrived soon after. The author judged that the whole service was affected.

   Service outageMemory pressure

   Root cause

   Capacity exhaustion

   Component

   Elasticsearch or OpenSearch

   Contributing factor

   No capacity planning

   Detection

   Customer success reported that the customer list in the admin screen would not open, customer inquiries followed, and the AWS console showed the OpenSearch cluster red.

   Remediation

   • Restarted the OpenSearch nodes, which cleared the outage temporarily in about 30 minutes
   • Created a new index with 9 shards, one per OpenSearch node, with no replicas and refresh turned off during the copy
   • Copied the old index into the new one with an asynchronous _reindex and compared document counts
   • Set number_of_replicas to 1 and refresh_interval back to 1s on the new index
   • Moved the customer_data alias used by the application to the new index
   • Ran a second _reindex to copy documents written to the old index after the first copy

   Lessons

   As the service scales, its non-functional parts need work, and the author wants to deal with them before they cause trouble.

   View case
3. IL-0290 GitLab.com's Redis Sidekiq cluster ran out of memory and lost quorum, causing 500 errors and a 12 million job backlog

   GitLab2026sev1

   Symptoms

   From 17:12 UTC on 12 May 2026, error rates rose at the same time across GitLab.com's web, Git, API and CI runner services. Users got frequent 500 errors, failed or stuck CI pipelines, authentication failures and unreliable Git access, and a Prometheus Alertmanager alert opened the incident at 17:15.

   High error rateMemory pressureThroughput degradation

   Trigger

   Earlier on 12 May, as a mitigation for a previous incident, INC-10084, a Sidekiq queueing apdex SLO violation on shard catchall-b, jobs of AuditEvents::AuditEventStreamingWorker were deferred. The deferred audit event jobs then piled up in the Redis Sidekiq cluster.

   Root cause

   Capacity exhaustion

   Component

   In-memory cache

   Contributing factor

   Change with a delayed effectPages not deliveredLimited debug access

   Detection

   A Prometheus Alertmanager alert reported the incident at 17:15:10 UTC, three minutes after impact began. A follow-up asks why the redis-sidekiq memory alert did not page, and the review says some signals did not reach the right on-call people.

   Remediation

   • Restarted and force-reset the two unresponsive Redis Sidekiq nodes
   • Stopped each Redis Sidekiq VM in turn and resized it to c2-standard-16, doubling memory and CPU
   • Re-enabled AuditEventStreamingWorker instead of deferring it

   Action items

   • Investigate why the redis-sidekiq memory alert did not page
   • Establish the root cause, failure mode and remediation status of the WebHookWorker poison pill
   • Give mitigations that disable safety limits an explicit owner and a deadline for re-evaluating them
   • Stop AuditEvents::AuditEventStreamingWorker from causing recurring incidents
   • Create a runbook for AuditEventStreamingWorker
   • Address PG connection pool saturation

   Lessons

   Broad degradation across web, Git, authentication, CI runners, Sidekiq and the internal API made it hard to settle on one cause and explain it clearly. The customer symptom of pipelines stuck in created was hard to measure in real time, and the dashboards that would have answered it were not obvious to on-call responders. Some alerts did not reach the right on-call people, and hosts that hung on login slowed diagnosis and made changes riskier. Redis-sidekiq under memory saturation forces choices between availability and data loss that are hard to make during an incident and call for guardrails and design changes.

   View case
4. IL-0435 Error rates rose as Aurora MySQL killed connections and declined queries to avoid OOM while load metrics stayed flat

   CyberAgent2026

   Symptoms

   An alert fired one day because an application's error rate had suddenly risen. The error log of the Aurora MySQL RDS instance for that time showed Aurora killing connections and declining queries to avoid running out of memory. The killed queries also made application latency worse.

   High error rateMemory pressure

   Trigger

   An application release added a batch job run every minute by a Kubernetes CronJob. Each run's short-lived pod connected to the database from a new pod IP.

   Root cause

   Capacity exhaustion

   Component

   DatabaseJobs and CronJobs

   Contributing factor

   Unreviewed defaultsChange with a delayed effect

   Detection

   An alert fired on the application's error rate.

   Remediation

   • Set performance_schema to manual management and capped two related Performance Schema parameters, unlimited by default, at 100 in the parameter group, then restarted the RDS instance

   Lessons

   When system alerts fire but the database's load metrics look calm, Performance Schema growth is worth suspecting. The two parameters that were capped are unlimited by default, so connections from ever-changing source addresses keep adding to the records. Capping them has a cost: once the cap is exceeded, per-host figures in Performance Insights are missing. Putting RDS Proxy between the application and Aurora MySQL to keep source hosts stable is another possible measure.

   View case
5. IL-0453 Hakuna API latency spiked every day during the ElastiCache Redis backup as memory filled and the backup went forkless

   Hyperconnect2022

   Symptoms

   The service API latency of Hyperconnect's Hakuna rose sharply at a certain time, periodically. The per-dependency latency analysis in the APM showed that a bottleneck in Redis caused it, but no deployment and no change in traffic pattern could have affected how Redis was used.

   High latencyMemory pressure

   Trigger

   The daily ElastiCache backup on the primary node ran while available memory was insufficient, so ElastiCache used a forkless backup that slowed user requests.

   Root cause

   Capacity exhaustion

   Component

   In-memory cacheDatabase backups and snapshots

   Contributing factor

   No alert on limit

   Detection

   A latency graph showed the service API latency spiking periodically, and the APM's per-dependency latency analysis pointed at Redis.

   Remediation

   • Moved the ElastiCache backup from the primary node to the secondary node
   • Changed the code so data stored without a TTL is stored with one
   • Replaced the Redis Hash CrudRepository usages with a CustomRepository built on RedisTemplate
   • Ran a script that deleted the unneeded data already stored, freeing 58 GB on 3 databases

   Lessons

   The team says its Redis operation and monitoring had gaps, and lists what to watch: use a relational database when data has relationships or must be searched by value; write data that must be durable to on-disk storage rather than Redis; keep available memory at 50% or more when ElastiCache snapshots must be taken; take care that O(n) commands such as DEL on a collection do not stall production Redis; and check regularly for unnecessary I/O between the application and the database. Spring Data Redis repositories, which manage secondary indexes in the framework, suit workloads that need no TTL and need fast queries.

   View case

By root cause1