Cases3 cases

1 to 3 of 3

1. IL-0581 29CM product display servers lost pods and saturated Netty event loops on the first day of the Igu Week sale

   MUSINSA2025

   Symptoms

   Shortly after 29CM's Igu Week sale began on 3 November 2025, the servers for the display domain, which serve search results and product listings, showed anomalies. Some pods went down and stopped taking traffic, and the remaining pods received more traffic than they could handle and saturated their Netty event loops.

   Probe failuresIntermittent connection failuresHigh latency

   Trigger

   Igu Week, 29CM's largest Black Friday sale and its busiest traffic of the year, raised Redis network traffic to about four times normal.

   Root cause

   Capacity exhaustion

   Component

   In-memory cacheLiveness and readiness probes

   Contributing factor

   Unknown limitMonitoring blind spotHealth check tied to a dependency

   Detection

   Anomalies on the display servers, pods going down and Netty event loops saturating, were the first sign. The Redis cause was found from Redis health check failure logs and then the Redis network metrics.

   Remediation

   • Scaled the Redis nodes up from cache.r7g.large to cache.r7g.2xlarge, about twice the baseline network bandwidth

   Action items

   • Extended the monitoring dashboard to show in real time whether network in and out bandwidth exceeds the baseline, with metrics such as aws.elasticache.network_bandwidth_out_allowance_exceeded and aws.elasticache.traffic_management_active
   • Added Datadog alerts that fire when network bandwidth crosses a threshold
   • Added a Caffeine local cache in front of Redis for data that changes rarely
   • Look into compressing cached data with Snappy to reduce Redis network bandwidth
   • Look into storing cached data as Protocol Buffers instead of JSON; measured on the cached Item Document data, Snappy and protobuf together are expected to cut the size by about 73%
   • Keep improving the architecture for traffic growth, building on the cache restructuring, stronger monitoring and read separation already applied

   Lessons

   CPU and memory metrics alone did not show the problem, so Redis needs network monitoring against the node's bandwidth baseline. Burst credits kept the nodes working for about two hours after traffic passed the baseline, which delayed the failure and made its cause slower to find.

   View case
2. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts

   Omio2020

   Symptoms

   For about six months, applications on Omio's Google Kubernetes Engine clusters stalled at random: they got stuck, stopped answering health check requests or lost network connections, and nothing the team looked at explained why.

   CPU throttlingProbe failuresHigh error rateHigh latencyIntermittent timeoutsIntermittent connection failures

   Trigger

   Multi-threaded workloads running with CPU limits on nodes whose kernel did not yet have the fix for a known CFS quota bug that causes unnecessary throttling.

   Root cause

   Software defect

   Component

   Container CPU limitLinux kernelkubelet

   Contributing factor

   Unknown limitMonitoring blind spotUnpatched known bug

   Detection

   Months of random stalls and failed health checks led the team to read the cgroup cpu.stat throttling counters inside the pods, which showed far more throttling than the limits should allow.

   Remediation

   • removed CPU limits until the kernel with the CFS bug fix was deployed across all clusters

   Action items

   • Deploy the kernel with the CFS quota bug fix across all clusters
   • Monitor utilisation, saturation and error metrics, API latency and error rates to confirm the change behaves as expected

   Lessons

   A CPU limit is a time quota per 100 ms period, not a share of cores, so a multi-threaded container can be stopped for most of each period long before it uses the cores it was given, and a kernel bug can make that worse. Throttling is invisible in standard Kubernetes metrics and has to be read from the cgroup counters. Removing limits trades away protection against a container that takes a whole node, which is only acceptable when runtimes are tuned.

   View case
3. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe

   NRE Labs2018

   Symptoms

   The daily NRE Labs statistics showed lesson activity collapsing, and kubectl could not reach the cluster: `The connection to the server 1.2.3.4:6443 was refused - did you specify the right host or port?`

   Control plane unresponsiveProbe failuresService outage

   Trigger

   Unknown. The liveness probe that kubeadm writes into the etcd static pod manifest (an `etcdctl get foo` against 127.0.0.1:2379, failureThreshold 8, timeoutSeconds 15) started failing. The author first suspected GCP maintenance the night before, but could not reproduce the probe failure by hand and does not know why it began.

   Root cause

   unknown

   Component

   etcdkubeletKubernetes API server

   Contributing factor

   Single point of failureUnreviewed defaultsHarmful manual actionIgnored early signalMonitoring blind spot

   Detection

   The author saw lesson activity drop in the daily NRE Labs statistics during his morning routine.

   Remediation

   • removed the liveness probe section from the kubeadm-generated etcd static pod manifest and applied it between etcd restarts

   Action items

   • Add monitoring, especially for the single-node master
   • Run a basic "down for maintenance" page in Google Cloud Functions that points to Twitter for updates, to redirect to during outages

   Lessons

   Because all of the infrastructure is managed as code, the cluster could be rebuilt quickly, but destroying it before finding the cause also destroyed the evidence. Monitoring matters, especially with a single-node master. Read the restart counts in kubectl output, not just the Running status. Remember the fundamentals instead of expecting the issue to be more complicated than it is, and do not let the urgency of an outage push you into hasty decisions. Put up a maintenance page instead of leaving the site failing halfway.

   View case

By root cause3