IL-0581 MUSINSA · 2025-11-03databaseawskubernetes
29CM product display servers lost pods and saturated Netty event loops on the first day of the Igu Week sale
29CM, part of MUSINSA, held Igu Week, its largest Black Friday sale, in winter 2025. The display servers used ElastiCache Redis on cache.r7g.large nodes with a baseline network bandwidth of 0.937Gbps, about 117MB per second. ElastiCache lets traffic go over an instance type's baseline for a while by spending burst credits, which build up while traffic stays under the baseline; when the credits run out, AWS throttles the node to the baseline. Normally Redis out traffic stayed at about 0.49Gbps (about 61MB/s), within the baseline. During the sale it rose to about 2.0Gbps (about 250MB/s).
From 19:00 the nodes ran above the baseline on burst. At 20:58 the credits built up over about two hours ran out and network throttling began. Redis connections and commands started to fail, the application's readiness probes failed, and many pods went down. Because the burst period hid the problem at first, finding the cause took longer. Once the cause was found, the team scaled the Redis nodes up from cache.r7g.large to cache.r7g.2xlarge, whose baseline of 1.875Gbps is about twice the network capacity, and the outage ended the same day.
Afterwards the team added ElastiCache network metrics and Datadog alerts, and moved data that rarely changes into a Caffeine local cache in front of Redis, so that requests go from the local cache to Redis to the database. Redis command counts and outgoing bytes dropped visibly. For the longer term the team is looking at compressing cached data with Snappy and storing it as Protocol Buffers instead of JSON, which it expects to cut the size of the cached item data by about 73%.
- Severity
- –
- Scope
- 29CM's main product display screens, including search results and product listings
- Users affected
- –
- Region
- –
- Time to detect
- –
- Time to resolve
- –
Symptoms
Shortly after 29CM's Igu Week sale began on 3 November 2025, the servers for the display domain, which serve search results and product listings, showed anomalies. Some pods went down and stopped taking traffic, and the remaining pods received more traffic than they could handle and saturated their Netty event loops.
The display servers run on Netty and Spring WebFlux, so the team first checked downstream delays and event loop delays caused by the traffic increase. Redis was checked too, but its CPU and memory metrics were all in the normal range, under 10%, so the team judged it an unlikely cause.
During the analysis, Redis health check failure logs stood out, and the team looked at the metrics again. The network metrics showed that Redis network out traffic, normally about 0.49Gbps, had risen to about 2.0Gbps at the peak, against a baseline bandwidth of 0.937Gbps for the cache.r7g.large nodes. Traffic had exceeded the baseline from 19:00, and at 20:58 Redis response delays and connection failures surged suddenly.
Symptom Probe failuresIntermittent connection failuresHigh latency
Root cause
Root cause Capacity exhaustion
- Trigger
- Igu Week, 29CM's largest Black Friday sale and its busiest traffic of the year, raised Redis network traffic to about four times normal.
Detection
Anomalies on the display servers, pods going down and Netty event loops saturating, were the first sign. The Redis cause was found from Redis health check failure logs and then the Redis network metrics.
Remediation
- Scaled the Redis nodes up from cache.r7g.large to cache.r7g.2xlarge, about twice the baseline network bandwidth
Action items
- Extended the monitoring dashboard to show in real time whether network in and out bandwidth exceeds the baseline, with metrics such as aws.elasticache.network_bandwidth_out_allowance_exceeded and aws.elasticache.traffic_management_active
- Added Datadog alerts that fire when network bandwidth crosses a threshold
- Added a Caffeine local cache in front of Redis for data that changes rarely
- Look into compressing cached data with Snappy to reduce Redis network bandwidth
- Look into storing cached data as Protocol Buffers instead of JSON; measured on the cached Item Document data, Snappy and protobuf together are expected to cut the size by about 73%
- Keep improving the architecture for traffic growth, building on the cache restructuring, stronger monitoring and read separation already applied
Lessons
CPU and memory metrics alone did not show the problem, so Redis needs network monitoring against the node's bandwidth baseline. Burst credits kept the nodes working for about two hours after traffic passed the baseline, which delayed the failure and made its cause slower to find.
Sources
Related cases
IL-0432AutoReserve notifications silently went unsent as Memorystore for Redis 3.2 rejected writes with OOM below its limitHello Inc. · 2026same root cause · shared In-memory cache
IL-0379InfluxDB Cloud queries in Azure West Europe slowed and timed out as storage pods fell behind and kept restartingInfluxData · 2026same root cause · shared Liveness and readiness probes
IL-0290GitLab.com's Redis Sidekiq cluster ran out of memory and lost quorum, causing 500 errors and a 12 million job backlogGitLab · 2026 · sev1same root cause · shared In-memory cache
IL-0100PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared In-memory cache
IL-0067Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restartBC Government · 2025 · sev1same root cause · shared In-memory cache
IL-0544Wasabi's database cache stopped responding when its queueing service filled up, and S3 failed in all regionsWasabi · 2024same root cause · shared In-memory cache
IL-0202A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1same root cause · shared In-memory cache
IL-0386After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodesincident.io · 2023same root cause · shared In-memory cache
IL-0453Hakuna API latency spiked every day during the ElastiCache Redis backup as memory filled and the backup went forklessHyperconnect · 2022same root cause · shared In-memory cache
IL-0212Many Slack users could not connect because a Vitess keyspace for channel membership was overloaded by cache-miss queriesSlack · 2022same root cause · shared In-memory cache
IL-0297Mezmo users could not sign in and log indexing lagged after the auth cache's Redis servers were OOM killedMezmo · 2019same root cause · shared In-memory cache
IL-0333A downsized RDS PostgreSQL database maxed out its CPU at US peak and failing ELB health checks took buildkite.com downBuildkite · 2016same root cause · shared Liveness and readiness probes
IL-0520logs.cloud.gov trapped users in a login redirect loop because its full Redis session store stopped saving sessionscloud.gov · 2016same root cause · shared In-memory cache
IL-0296Transloadit's us-east Redis cluster lost queued jobs and heartbeats, raising WORKER_JOB_ERROR errors for 81 customersTransloadit · 2016same root cause · shared In-memory cache
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0581
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0581/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.