Incident Library

IL-0453 Hyperconnect · 2022-12-12 (published)databaseaws

Hakuna API latency spiked every day during the ElastiCache Redis backup as memory filled and the backup went forkless

Hakuna's ElastiCache Redis uses Auto-failover for availability. The primary node was busy with service requests as well as the backup, while the secondary node only replicated the primary and was mostly idle. Running the backup on the secondary can make the backup inconsistent because changes reach the secondary late, but a once-a-day backup is already inconsistent to some degree, so the team accepted that and moved the backup to the secondary node, which removed the bottleneck caused by the backup.

The memory growth came from how Hakuna used Spring Data Redis repositories through the CrudRepository interface. To support count and query derivation on a key-value store, Spring Data Redis keeps a set of all keys per domain class and, for fields annotated with @Indexed, a secondary index keyed domain_class:field_name:value. When data gets a TTL through @TimeToLive or @RedisHash(timeToLive = 1), the data itself expires, but its entry in the key set stays, so the set keeps growing and count returns wrong numbers. Keeping the sets consistent requires subscribing to Redis keyspace notifications with @EnableRedisRepositories(enableKeyspaceEvents = ON_STARTUP), in which case each save also writes a copy under domain_class:ID:phantom so the expired data can be restored into a RedisKeyExpiredEvent. The team decided that accessing data through RedisTemplate was better, because the business used few of the CrudRepository methods and each save carried hidden extra commands.

The team first changed the code so that data stored without a TTL is stored with one, then reimplemented the places that used the Redis Hash CrudRepository as a CustomRepository on RedisTemplate, and then ran a script to delete the unneeded data that had piled up. This freed 58 GB of memory in total on 3 databases, and memory usage stopped growing.

Severity
–
Scope
Hakuna service API latency spiked during the daily ElastiCache backup
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

The service API latency of Hyperconnect's Hakuna rose sharply at a certain time, periodically. The per-dependency latency analysis in the APM showed that a bottleneck in Redis caused it, but no deployment and no change in traffic pattern could have affected how Redis was used.

The team then found that the spikes coincided with the AWS ElastiCache backup time. Hakuna had run ElastiCache backups periodically for a long time, so the team looked into why they started to hurt only now. When available memory is not sufficient, ElastiCache chooses a forkless backup strategy, and a forkless backup degrades the latency and throughput of user requests. Further monitoring showed that Redis memory usage had been rising steadily, which had left too little available memory.

To find what was using the memory, the team suspected data stored without a TTL and kept forever. A script walked all keys with scan and wrote each key's name, TTL, data type and memory use to a file. Comparing that list with the Hakuna code turned up sets with the key pattern domain_class and sets with the key pattern domain_class:ID:phantom that together used more than 10 GB of memory.

Symptom High latencyMemory pressure

Root cause

Root cause Capacity exhaustion

Component In-memory cacheDatabase backups and snapshots
Contributing factor No alert on limit
Trigger
The daily ElastiCache backup on the primary node ran while available memory was insufficient, so ElastiCache used a forkless backup that slowed user requests.

Detection

A latency graph showed the service API latency spiking periodically, and the APM's per-dependency latency analysis pointed at Redis.

Remediation

Action items

–

Lessons

The team says its Redis operation and monitoring had gaps, and lists what to watch: use a relational database when data has relationships or must be searched by value; write data that must be durable to on-disk storage rather than Redis; keep available memory at 50% or more when ElastiCache snapshots must be taken; take care that O(n) commands such as DEL on a collection do not stall production Redis; and check regularly for unnecessary I/O between the application and the database. Spring Data Redis repositories, which manage secondary indexes in the framework, suit workloads that need no TTL and need fast queries.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, In-memory cache, Database backups and snapshots among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.