Cases2 cases

1 to 2 of 2

1. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks

   AdGuard VPN2025

   Symptoms

   The day began with an alert that several instances of the Redis cluster were running out of disk space. That part was fixed quickly. The outage came an hour later: right after the Redis StatefulSet was deleted and recreated at 8:10 UTC, Kubernetes restarted every Redis pod within about five minutes, and problems started to appear.

   Service outageOOMKilledProbe failuresHigh latency

   Trigger

   To make Git match the volumes that had been resized by hand, the Redis StatefulSet was deleted with orphan (non-cascading) deletion and recreated with the new storage template; Kubernetes reconciled the new spec by restarting the entire Redis cluster.

   Root cause

   Configuration change

   Component

   Persistent volumes and stateful setsIn-memory cacheContainer memory limit

   Contributing factor

   No health probesUnverified assumptionSingle point of failureRetry stormNo headroom

   Detection

   An alert about Redis instances running out of disk space started the work; the outage itself showed up as backend errors about Redis connections right after the StatefulSet was recreated.

   Remediation

   • made the Redis masters forget obsolete node identities, reset the affected replicas and rejoined them, then waited for replication to catch up
   • identified the component that was actually running out of memory, scaled the backend service and raised memory limits for that component
   • temporarily relaxed probes so backend instances took traffic immediately instead of flapping between ready and not ready
   • reverted the temporary workarounds at 10:00 UTC

   Action items

   • Add startup probes so a Redis pod is not marked ready until its dataset is loaded
   • Build an own control plane for Redis that handles restarted pods stuck to an old master address
   • Reduce dependence on the single Redis cluster: move some important data to other stores, such as a separate Redis cluster per server, and use Kafka for some backend components

   Lessons

   The disk resize was not the problem; the cleanup afterwards was. An immutable field pushes operators toward deleting and recreating the controller, and that can set off a full reconcile. Startup and readiness probes are what slow a rolling restart down to a safe pace; without them a distributed store restarts in the most fragile way. Recovery plans must expect a cluster to heal slowly, and dependent services need to degrade gracefully while it does.

   View case
2. IL-0581 29CM product display servers lost pods and saturated Netty event loops on the first day of the Igu Week sale

   MUSINSA2025

   Symptoms

   Shortly after 29CM's Igu Week sale began on 3 November 2025, the servers for the display domain, which serve search results and product listings, showed anomalies. Some pods went down and stopped taking traffic, and the remaining pods received more traffic than they could handle and saturated their Netty event loops.

   Probe failuresIntermittent connection failuresHigh latency

   Trigger

   Igu Week, 29CM's largest Black Friday sale and its busiest traffic of the year, raised Redis network traffic to about four times normal.

   Root cause

   Capacity exhaustion

   Component

   In-memory cacheLiveness and readiness probes

   Contributing factor

   Unknown limitMonitoring blind spotHealth check tied to a dependency

   Detection

   Anomalies on the display servers, pods going down and Netty event loops saturating, were the first sign. The Redis cause was found from Redis health check failure logs and then the Redis network metrics.

   Remediation

   • Scaled the Redis nodes up from cache.r7g.large to cache.r7g.2xlarge, about twice the baseline network bandwidth

   Action items

   • Extended the monitoring dashboard to show in real time whether network in and out bandwidth exceeds the baseline, with metrics such as aws.elasticache.network_bandwidth_out_allowance_exceeded and aws.elasticache.traffic_management_active
   • Added Datadog alerts that fire when network bandwidth crosses a threshold
   • Added a Caffeine local cache in front of Redis for data that changes rarely
   • Look into compressing cached data with Snappy to reduce Redis network bandwidth
   • Look into storing cached data as Protocol Buffers instead of JSON; measured on the cached Item Document data, Snappy and protobuf together are expected to cut the size by about 73%
   • Keep improving the architecture for traffic growth, building on the cache restructuring, stronger monitoring and read separation already applied

   Lessons

   CPU and memory metrics alone did not show the problem, so Redis needs network monitoring against the node's bandwidth baseline. Burst credits kept the nodes working for about two hours after traffic passed the baseline, which delayed the failure and made its cause slower to find.

   View case

By root cause2