Cases1 case

1 to 1 of 1

1. IL-0453 Hakuna API latency spiked every day during the ElastiCache Redis backup as memory filled and the backup went forkless

   Hyperconnect2022

   Symptoms

   The service API latency of Hyperconnect's Hakuna rose sharply at a certain time, periodically. The per-dependency latency analysis in the APM showed that a bottleneck in Redis caused it, but no deployment and no change in traffic pattern could have affected how Redis was used.

   High latencyMemory pressure

   Trigger

   The daily ElastiCache backup on the primary node ran while available memory was insufficient, so ElastiCache used a forkless backup that slowed user requests.

   Root cause

   Capacity exhaustion

   Component

   In-memory cacheDatabase backups and snapshots

   Contributing factor

   No alert on limit

   Detection

   A latency graph showed the service API latency spiking periodically, and the APM's per-dependency latency analysis pointed at Redis.

   Remediation

   • Moved the ElastiCache backup from the primary node to the secondary node
   • Changed the code so data stored without a TTL is stored with one
   • Replaced the Redis Hash CrudRepository usages with a CustomRepository built on RedisTemplate
   • Ran a script that deleted the unneeded data already stored, freeing 58 GB on 3 databases

   Lessons

   The team says its Redis operation and monitoring had gaps, and lists what to watch: use a relational database when data has relationships or must be searched by value; write data that must be durable to on-disk storage rather than Redis; keep available memory at 50% or more when ElastiCache snapshots must be taken; take care that O(n) commands such as DEL on a collection do not stall production Redis; and check regularly for unnecessary I/O between the application and the database. Spring Data Redis repositories, which manage secondary indexes in the framework, suit workloads that need no TTL and need fast queries.

   View case

By root cause1