IL-0181 Kevin · 2026-02-19 (published)kubernetes
The product service pods ran with a CPU request of 500m and a CPU limit of 1 core. When the sudden traffic hit the product lookup API, the containers reached that limit and the Linux CFS (Completely Fair Scheduler) held back their CPU use, so requests were not processed in time even though the pods looked healthy. The author had believed the resource settings could absorb the event traffic the team expected, and a single external post pushed traffic beyond that.
The author first raised minReplicas of the HorizontalPodAutoscaler for my-service-prd from 5 to 10 (maxReplicas 20, target average CPU utilization 70%). With the same traffic spread over 10 pods, per-pod CPU use dropped and response times returned to normal. The author then raised the container CPU limit from 1 core to 2 cores, noting that the base setting had been 3 cores, and plans to keep adjusting the limit through monitoring so that pods do not take too much node capacity.
An e-commerce platform suddenly became slow to respond, worst on the product lookup page, and users began reporting that the page would not open.
The Kubernetes dashboard showed every pod Running with no restarts. The CPU usage of the affected service's namespace was close to 100%. The pods were not restarted the way an OOMKill would restart them and the logs showed no clear error; looking at the pod list alone, the author did not see the cause. Only the CPU usage metrics showed that the containers were being throttled.
Symptom CPU throttlingHigh latency
Root cause Capacity exhaustion
Users reported that the product page would not open. The dashboard showed all pods Running, and the namespace CPU usage near 100% and the CPU usage metrics pointed to throttling.
CPU throttling is hard to notice: pods stay Running, nothing restarts and the logs show no clear error, so a pod list does not reveal it and CPU usage metrics do.
Settings that had handled expected event traffic became the bottleneck as soon as one external post multiplied the traffic. The author concludes that the team needs more preparation for traffic pattern changes and more thought about resource settings.
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared Horizontal pod autoscaler
IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation이언철 · 2024same root cause · shared Horizontal pod autoscaler
IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percentTemporal · 2023same root cause · shared Container CPU limit
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.