Cases2 cases

1 to 2 of 2

1. IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hours

   Microsoft Azure2026

   Symptoms

   From 10:03 UTC on 29 September 2026, requests to Azure OpenAI Service, Foundry Agent Service, Foundry Models and Azure AI services in Sweden Central failed intermittently, took longer, or returned HTTP 500 and 503 errors. At 10:17 UTC service monitoring raised an alert on elevated failure rates for Azure OpenAI Service requests in the region.

   High error rateHigh latencyIntermittent connection failuresIntermittent timeoutsProbe failuresOutbound connection exhaustion

   Trigger

   At about 09:38 UTC a planned deployment of an internal service in Sweden Central cleared that service's caches during the region's peak traffic hours.

   Root cause

   Faulty deployment

   Component

   In-memory cacheApplication database

   Contributing factor

   Change during peak hoursRetry stormAutomation amplificationMonitoring blind spotNo runbookHealth check tied to a dependency

   Detection

   Service monitoring detected elevated failure rates for Azure OpenAI Service requests in the region at 10:17 UTC and alerted the engineering teams; engineers were engaged at 10:28 UTC.

   Remediation

   • reconfigured the backend service's automated health check so it no longer depended on remote connectivity, which stopped the restart loop

   Action items

   • Reconfigure the backend service's automated health check so it no longer depends on connectivity to its database layer
   • Add monitoring and alerting on the load and latency of the caching layer used by the backend service
   • Update troubleshooting and procedural guidance for connection exhaustion scenarios for on-call engineers
   • Add throttling and rate limiting to protect the backend service from sudden traffic increases (estimated October 2026)
   • Increase the resource capacity of the caching layer used by the backend service across regions (estimated October 2026)
   • Improve diagnostic metrics and automated detection to identify such issues more quickly (estimated October 2026)
   • Partition the caching layer used by the backend service to increase its throughput (estimated December 2026)

   Lessons

   A health check that depends on a remote dependency turns a dependency problem into a restart loop: instances that cannot reach the database are killed, the remaining ones take more load and retries, and adding instances or resources does not help until the check itself is changed. Clearing a cache during peak traffic can multiply load on the backend behind it, especially when cache-read defects let lookups bypass the cache. Connection exhaustion was hard to see because the existing network and diagnostic metrics did not show it.

   View case
2. IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse

   RebelMouse2025

   Symptoms

   At 13:42 UTC on 19 May 2025, RebelMouse's monitoring tools reported a higher error rate and degraded performance across services. A routine reroll of the EKS nodes had started 27 minutes earlier, at 13:15 UTC.

   DNS resolution failuresHigh latencyHigh error rateService outage

   Trigger

   At 13:15 UTC RebelMouse started its regular reroll of EKS nodes to apply security patches. The planned, automatic termination of an EC2 instance terminated one of the CoreDNS pods.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSNode pool

   Contributing factor

   Change during peak hours

   Detection

   Monitoring tools detected an increased error rate and degraded service performance at 13:42 UTC, and the tech team started investigating at 13:44 UTC.

   Remediation

   • rescheduled the affected CoreDNS pod onto a new EC2 instance
   • scaled up the CoreDNS deployment to more replicas so DNS load was spread evenly across all pods

   Action items

   • Add scale-in protection so CoreDNS pods are not terminated unexpectedly during peak load
   • Review the CoreDNS configuration so that vertical scaling of the pods works
   • Hold regular capacity reviews so the CoreDNS deployment is provisioned for peak load

   View case

By root cause2