Cases3 cases

1 to 3 of 3

1. IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hours

   Microsoft Azure2026

   Symptoms

   From 10:03 UTC on 29 September 2026, requests to Azure OpenAI Service, Foundry Agent Service, Foundry Models and Azure AI services in Sweden Central failed intermittently, took longer, or returned HTTP 500 and 503 errors. At 10:17 UTC service monitoring raised an alert on elevated failure rates for Azure OpenAI Service requests in the region.

   High error rateHigh latencyIntermittent connection failuresIntermittent timeoutsProbe failuresOutbound connection exhaustion

   Trigger

   At about 09:38 UTC a planned deployment of an internal service in Sweden Central cleared that service's caches during the region's peak traffic hours.

   Root cause

   Faulty deployment

   Component

   In-memory cacheApplication database

   Contributing factor

   Change during peak hoursRetry stormAutomation amplificationMonitoring blind spotNo runbookHealth check tied to a dependency

   Detection

   Service monitoring detected elevated failure rates for Azure OpenAI Service requests in the region at 10:17 UTC and alerted the engineering teams; engineers were engaged at 10:28 UTC.

   Remediation

   • reconfigured the backend service's automated health check so it no longer depended on remote connectivity, which stopped the restart loop

   Action items

   • Reconfigure the backend service's automated health check so it no longer depends on connectivity to its database layer
   • Add monitoring and alerting on the load and latency of the caching layer used by the backend service
   • Update troubleshooting and procedural guidance for connection exhaustion scenarios for on-call engineers
   • Add throttling and rate limiting to protect the backend service from sudden traffic increases (estimated October 2026)
   • Increase the resource capacity of the caching layer used by the backend service across regions (estimated October 2026)
   • Improve diagnostic metrics and automated detection to identify such issues more quickly (estimated October 2026)
   • Partition the caching layer used by the backend service to increase its throughput (estimated December 2026)

   Lessons

   A health check that depends on a remote dependency turns a dependency problem into a restart loop: instances that cannot reach the database are killed, the remaining ones take more load and retries, and adding instances or resources does not help until the check itself is changed. Clearing a cache during peak traffic can multiply load on the backend behind it, especially when cache-read defects let lookups bypass the cache. Connection exhaustion was hard to see because the existing network and diagnostic metrics did not show it.

   View case
2. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts

   DFDS2022

   Symptoms

   On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.

   Node NotReadyUpgrade stallIntermittent timeoutsIntermittent connection failures

   Trigger

   At 07:30 UTC on 2022-10-31 the daily node rollover replaced the existing nodes with nodes built from AMI v20221027, the latest AWS EKS node AMI, released on 2022-10-29 and later recalled because of a kernel issue.

   Root cause

   Software defect

   Component

   Node operating system imageLinux kernelkubeletAWS VPC CNINode launch templateTerraform

   Contributing factor

   No staged rolloutAutomation amplificationOverlapping failuresMonitoring blind spotNo health probes

   Detection

   The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.

   Remediation

   • cordoned, drained and deleted 15 stuck nodes one at a time to complete the vpc-cni upgrade
   • changed the Terraform infrastructure-modules so a specific AMI could be set
   • rolled the EC2 Auto Scaling group launch configuration back to the previous known good AMI through the pipeline

   Action items

   • Reach out to teams with suggestions about readiness and liveness probes
   • Create a dashboard for readiness and liveness probe coverage
   • Write a wiki article on readiness and liveness probes and their usefulness
   • Modify infrastructure-modules to allow specifying a particular AMI
   • Schedule a discussion on whether to take the latest AMI immediately and how to test new AMIs before they are deployed
   • Implement alerting on abnormal NodeNotReady events and on increased socket use
   • Create a learning path for Kubernetes within the team
   • Collect logs from nodes centrally
   • Create a Prometheus exporter that retrieves the latest available AMI, with a dashboard and alerts

   Lessons

   When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters. Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.

   View case
3. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts

   Omio2020

   Symptoms

   For about six months, applications on Omio's Google Kubernetes Engine clusters stalled at random: they got stuck, stopped answering health check requests or lost network connections, and nothing the team looked at explained why.

   CPU throttlingProbe failuresHigh error rateHigh latencyIntermittent timeoutsIntermittent connection failures

   Trigger

   Multi-threaded workloads running with CPU limits on nodes whose kernel did not yet have the fix for a known CFS quota bug that causes unnecessary throttling.

   Root cause

   Software defect

   Component

   Container CPU limitLinux kernelkubelet

   Contributing factor

   Unknown limitMonitoring blind spotUnpatched known bug

   Detection

   Months of random stalls and failed health checks led the team to read the cgroup cpu.stat throttling counters inside the pods, which showed far more throttling than the limits should allow.

   Remediation

   • removed CPU limits until the kernel with the CFS bug fix was deployed across all clusters

   Action items

   • Deploy the kernel with the CFS quota bug fix across all clusters
   • Monitor utilisation, saturation and error metrics, API latency and error rates to confirm the change behaves as expected

   Lessons

   A CPU limit is a time quota per 100 ms period, not a share of cores, so a multi-threaded container can be stopped for most of each period long before it uses the cores it was given, and a kernel bug can make that worse. Throttling is invisible in standard Kubernetes metrics and has to be read from the cgroup counters. Removing limits trades away protection against a container that takes a whole node, which is only acceptable when runtimes are tuned.

   View case

By root cause2