Incident Library
Applied filters Root causeFaulty deploymentOrganisationMicrosoft AzureClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hours
    Microsoft Azure2026
    Symptoms

    From 10:03 UTC on 29 September 2026, requests to Azure OpenAI Service, Foundry Agent Service, Foundry Models and Azure AI services in Sweden Central failed intermittently, took longer, or returned HTTP 500 and 503 errors. At 10:17 UTC service monitoring raised an alert on elevated failure rates for Azure OpenAI Service requests in the region.

    High error rateHigh latencyIntermittent connection failuresIntermittent timeoutsProbe failuresOutbound connection exhaustion
    Trigger

    At about 09:38 UTC a planned deployment of an internal service in Sweden Central cleared that service's caches during the region's peak traffic hours.

    Root cause
    Faulty deployment
    Component
    In-memory cacheApplication database
    Contributing factor
    Change during peak hoursRetry stormAutomation amplificationMonitoring blind spotNo runbookHealth check tied to a dependency
    Detection

    Service monitoring detected elevated failure rates for Azure OpenAI Service requests in the region at 10:17 UTC and alerted the engineering teams; engineers were engaged at 10:28 UTC.

    Remediation
    • reconfigured the backend service's automated health check so it no longer depended on remote connectivity, which stopped the restart loop
    Action items
    • Reconfigure the backend service's automated health check so it no longer depends on connectivity to its database layer
    • Add monitoring and alerting on the load and latency of the caching layer used by the backend service
    • Update troubleshooting and procedural guidance for connection exhaustion scenarios for on-call engineers
    • Add throttling and rate limiting to protect the backend service from sudden traffic increases (estimated October 2026)
    • Increase the resource capacity of the caching layer used by the backend service across regions (estimated October 2026)
    • Improve diagnostic metrics and automated detection to identify such issues more quickly (estimated October 2026)
    • Partition the caching layer used by the backend service to increase its throughput (estimated December 2026)
    Lessons

    A health check that depends on a remote dependency turns a dependency problem into a restart loop: instances that cannot reach the database are killed, the remaining ones take more load and retries, and adding instances or resources does not help until the check itself is changed. Clearing a cache during peak traffic can multiply load on the backend behind it, especially when cache-read defects let lookups bypass the cache. Connection exhaustion was hard to see because the existing network and diagnostic metrics did not show it.

    View case
  2. IL-0149 Upgrades of AKS clusters on Virtual Machine Scale Sets failed with a 'platform image is not available' error after a service update left a backend service unhealthy
    Microsoft Azure2019
    Symptoms

    From 07:20 UTC on 28 September 2019, customers running Azure Kubernetes Service clusters on Virtual Machine Scale Sets (VMSS) got failures when they tried any cluster upgrade. The error read: "The platform image 'microsoft-aks:aks:aks-ubuntu-1604-201909:2019.08.21' is not available. Verify that all fields in the storage profile are correct." The issue title calls the error PlatformImageNotFound.

    Upgrade stall
    Trigger

    A recent service update affected instances of a backend service, which became unhealthy and stopped requests from completing.

    Root cause
    Faulty deployment
    Component
    Node operating system imageNode pool
    Contributing factor
    –
    Detection

    Customers received failure messages when they tried to upgrade their clusters.

    Remediation
    • published az vmss update commands that set the scale set's image version to 2019.09.25 and SKU to aks-ubuntu-1604-201909, after which the upgrade could be run again
    • mitigated affected clusters by hand (AKS and VMSS teams)
    • rolled out hot fixes to all regions, completed by 15 October 2019
    View case
Check your cluster for this

These 2 cases involve In-memory cache and Application database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment2 cases

Component
In-memory cache1Application database1Node operating system image1Node pool1
Contributing factor
Automation amplification1Change during peak hours1Monitoring blind spot1No runbook1Health check tied to a dependency1Retry storm1
Organisation
Microsoft Azure2
  1. IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hours
    Microsoft Azure2026
  2. IL-0149 Upgrades of AKS clusters on Virtual Machine Scale Sets failed with a 'platform image is not available' error after a service update left a backend service unhealthy
    Microsoft Azure2019