Cases9 cases

1 to 5 of 9

1. IL-0181 A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed Running

   Kevin2026

   Symptoms

   An e-commerce platform suddenly became slow to respond, worst on the product lookup page, and users began reporting that the page would not open.

   CPU throttlingHigh latency

   Trigger

   A post on the external community Ppomppu linked to the platform and brought a large number of users in a short time, who called the product lookup API heavily.

   Root cause

   Capacity exhaustion

   Component

   Container CPU limitHorizontal pod autoscaler

   Contributing factor

   Monitoring blind spot

   Detection

   Users reported that the product page would not open. The dashboard showed all pods Running, and the namespace CPU usage near 100% and the CPU usage metrics pointed to throttling.

   Remediation

   • raised the HPA minReplicas from 5 to 10
   • raised the container CPU limit from 1 core to 2 cores

   Action items

   • Keep adjusting the CPU limit to a suitable level through monitoring

   Lessons

   CPU throttling is hard to notice: pods stay Running, nothing restarts and the logs show no clear error, so a pod list does not reveal it and CPU usage metrics do. Settings that had handled expected event traffic became the bottleneck as soon as one external post multiplied the traffic. The author concludes that the team needs more preparation for traffic pattern changes and more thought about resource settings.

   View case
2. IL-0581 29CM product display servers lost pods and saturated Netty event loops on the first day of the Igu Week sale

   MUSINSA2025

   Symptoms

   Shortly after 29CM's Igu Week sale began on 3 November 2025, the servers for the display domain, which serve search results and product listings, showed anomalies. Some pods went down and stopped taking traffic, and the remaining pods received more traffic than they could handle and saturated their Netty event loops.

   Probe failuresIntermittent connection failuresHigh latency

   Trigger

   Igu Week, 29CM's largest Black Friday sale and its busiest traffic of the year, raised Redis network traffic to about four times normal.

   Root cause

   Capacity exhaustion

   Component

   In-memory cacheLiveness and readiness probes

   Contributing factor

   Unknown limitMonitoring blind spotHealth check tied to a dependency

   Detection

   Anomalies on the display servers, pods going down and Netty event loops saturating, were the first sign. The Redis cause was found from Redis health check failure logs and then the Redis network metrics.

   Remediation

   • Scaled the Redis nodes up from cache.r7g.large to cache.r7g.2xlarge, about twice the baseline network bandwidth

   Action items

   • Extended the monitoring dashboard to show in real time whether network in and out bandwidth exceeds the baseline, with metrics such as aws.elasticache.network_bandwidth_out_allowance_exceeded and aws.elasticache.traffic_management_active
   • Added Datadog alerts that fire when network bandwidth crosses a threshold
   • Added a Caffeine local cache in front of Redis for data that changes rarely
   • Look into compressing cached data with Snappy to reduce Redis network bandwidth
   • Look into storing cached data as Protocol Buffers instead of JSON; measured on the cached Item Document data, Snappy and protobuf together are expected to cut the size by about 73%
   • Keep improving the architecture for traffic growth, building on the cache restructuring, stronger monitoring and read separation already applied

   Lessons

   CPU and memory metrics alone did not show the problem, so Redis needs network monitoring against the node's bandwidth baseline. Burst credits kept the nodes working for about two hours after traffic passed the baseline, which delayed the failure and made its cause slower to find.

   View case
3. IL-0413 LLM inference engines timed out or ran out of resources under load in about 60% of 156 severe incidents at Microsoft

   Microsoft2025

   Symptoms

   In Microsoft's cloud LLM inference service, the inference engines themselves failed in 94 of the 156 high-severity incidents from April to June 2025 (≈60%). Requests timed out or ran into resource exhaustion under load, and callers got HTTP 408, 429 and 5xx errors.

   Intermittent timeoutsHigh error rateHigh latency

   Trigger

   Bursty load, traffic with unusually high sampling parameters and long-running requests pushed the inference engines into resource exhaustion and timeouts.

   Root cause

   Capacity exhaustion

   Component

   Model inference server

   Contributing factor

   Monitoring blind spot

   Detection

   Across the 156 incidents, 115 (≈74%) were auto-detected, meaning health probes or DRI pipelines alerted before customers reported the problem; the others were not.

   Remediation

   • Enabled dynamic pushback at the load balancer with caps on pending tokens and requests
   • Tuned request batching and GPU/CPU replica counts
   • Improved gRPC backoff and error mapping
   • Limited sampling parameters and throttled per client
   • Enabled streaming, tuned request batching and capped pending tokens against engine connection timeouts on long-running requests
   • Increased capacity, routed traffic to healthy endpoints and restarted nodes, the actions applied mostly to inference engine failures

   Action items

   • Optimise serving for long-running requests and high sampling regimes, with connection liveness and GPU capacity-aware routing as the main levers
   • Expand health-probe-driven failover and GPU capacity-aware routing so traffic moves off degraded endpoints on its own
   • Forecast token and throughput growth and automate capacity increase decisions
   • Deploy adaptive alerting and expand engine-level telemetry to close detection gaps

   Lessons

   Hotfixes restore correctness but lengthen time to mitigate, while traffic routing and node rebalancing mitigate faster under overload and timeout patterns. Even with AIOps tooling a meaningful share of incidents still needs manual detection or judgement, so health-probe-driven traffic routing, restarts and capacity increases are the decisions to automate first.

   View case
4. IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percent

   Temporal2023

   Symptoms

   Request latency from the Temporal Server History service, the service that starts workflow executions and records their progress, was higher and less steady than it should be on a self-hosted Kubernetes deployment. The CPU dashboard showed the history pods using less than 60 percent of their CPU, yet the same pods were being throttled.

   CPU throttlingHigh latency

   Trigger

   Normal load on History pods that had a CPU limit of 2 cores on 8-core nodes, with the Go runtime's GOMAXPROCS left at its default of one per node core.

   Root cause

   Capacity exhaustion

   Component

   Container CPU limitLanguage runtime settingsPrometheus

   Contributing factor

   Unreviewed defaultsMonitoring blind spot

   Detection

   A Prometheus query for CPU throttling showed the history pods throttled although their CPU usage read under 60 percent.

   Remediation

   • set the GOMAXPROCS environment variable to 2 to match the 2-core CPU limit

   Action items

   • Set GOMAXPROCS automatically from the Kubernetes CPU limit, from Temporal release 1.21.0

   Lessons

   A CPU usage graph averaged over 15 seconds or more can sit far below 100 percent while a container is being throttled, so throttling needs its own metric. A Go program in a container should have GOMAXPROCS matched to its CPU limit, or to its CPU request when it has no limit. The author recommends not using CPU limits at all.

   View case
5. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS

   Datadog2022

   Symptoms

   From September 2021, each rollout of an update to Datadog's metrics query service, which serves real-time and historical metric data to the web frontend and to monitor evaluation, came with a rise in errors. Clients retried failed queries, so few errors reached users, but the retries slowed dashboards and monitors. The service's logs showed DNS errors stopping it from connecting to the data stores it depends on.

   High error rateDNS resolution failuresHigh latencyOOMKilledPacket lossStale endpoint routing

   Trigger

   Rollouts of the metrics query service, which delete the old pods. The client behaviour behind the failures came from an August 2021 rollout of the metrics query clients that applied round_robin as the new default gRPC load balancing policy.

   Root cause

   Configuration change

   Component

   conntrack tableCiliumLinux kernelCoreDNSClient-side DNS resolver and cacheexternal-dnsHTTP and RPC client library

   Contributing factor

   Change with a delayed effectRetry stormUnknown limitMonitoring blind spot

   Detection

   Errors rose during rollouts of the metrics query service, and the service's logs showed DNS errors when connecting to its data stores.

   Remediation

   • Raised the memory of the node-local-dns pods from 64 MB to 256 MB, which stopped the OOM kills but not the DNS errors
   • Tested larger instance types and confirmed that scaling up the service's infrastructure resolved the errors
   • Removed the custom gRPC reconnect parameters from the clients to restore the defaults

   Action items

   • Contribute a change to Cilium that adds an unreachable route for a deleted pod's IP so clients get an ICMP error

   Lessons

   The errors looked like DNS but were not. ENA metrics and VPC Flow Logs gave the visibility into VPC connection tracking that host metrics did not, and Cilium, Kubernetes and AWS networking interact in ways that can make reverse path filtering drop packets it should allow. DNS propagation time depends on many settings. The causes were changes Datadog had made itself, to work around a dependency bug, to resolve earlier incidents and to override default reconnect parameters, which looked safe at the time and showed their effects much later, when the link was hard to see.

   View case

By root cause4