Cases16 cases

1 to 5 of 16

1. IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hours

   Microsoft Azure2026

   Symptoms

   From 10:03 UTC on 29 September 2026, requests to Azure OpenAI Service, Foundry Agent Service, Foundry Models and Azure AI services in Sweden Central failed intermittently, took longer, or returned HTTP 500 and 503 errors. At 10:17 UTC service monitoring raised an alert on elevated failure rates for Azure OpenAI Service requests in the region.

   High error rateHigh latencyIntermittent connection failuresIntermittent timeoutsProbe failuresOutbound connection exhaustion

   Trigger

   At about 09:38 UTC a planned deployment of an internal service in Sweden Central cleared that service's caches during the region's peak traffic hours.

   Root cause

   Faulty deployment

   Component

   In-memory cacheApplication database

   Contributing factor

   Change during peak hoursRetry stormAutomation amplificationMonitoring blind spotNo runbookHealth check tied to a dependency

   Detection

   Service monitoring detected elevated failure rates for Azure OpenAI Service requests in the region at 10:17 UTC and alerted the engineering teams; engineers were engaged at 10:28 UTC.

   Remediation

   • reconfigured the backend service's automated health check so it no longer depended on remote connectivity, which stopped the restart loop

   Action items

   • Reconfigure the backend service's automated health check so it no longer depends on connectivity to its database layer
   • Add monitoring and alerting on the load and latency of the caching layer used by the backend service
   • Update troubleshooting and procedural guidance for connection exhaustion scenarios for on-call engineers
   • Add throttling and rate limiting to protect the backend service from sudden traffic increases (estimated October 2026)
   • Increase the resource capacity of the caching layer used by the backend service across regions (estimated October 2026)
   • Improve diagnostic metrics and automated detection to identify such issues more quickly (estimated October 2026)
   • Partition the caching layer used by the backend service to increase its throughput (estimated December 2026)

   Lessons

   A health check that depends on a remote dependency turns a dependency problem into a restart loop: instances that cannot reach the database are killed, the remaining ones take more load and retries, and adding instances or resources does not help until the check itself is changed. Clearing a cache during peak traffic can multiply load on the backend behind it, especially when cache-read defects let lookups bypass the cache. Connection exhaustion was hard to see because the existing network and diagnostic metrics did not show it.

   View case
2. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks

   AdGuard VPN2025

   Symptoms

   The day began with an alert that several instances of the Redis cluster were running out of disk space. That part was fixed quickly. The outage came an hour later: right after the Redis StatefulSet was deleted and recreated at 8:10 UTC, Kubernetes restarted every Redis pod within about five minutes, and problems started to appear.

   Service outageOOMKilledProbe failuresHigh latency

   Trigger

   To make Git match the volumes that had been resized by hand, the Redis StatefulSet was deleted with orphan (non-cascading) deletion and recreated with the new storage template; Kubernetes reconciled the new spec by restarting the entire Redis cluster.

   Root cause

   Configuration change

   Component

   Persistent volumes and stateful setsIn-memory cacheContainer memory limit

   Contributing factor

   No health probesUnverified assumptionSingle point of failureRetry stormNo headroom

   Detection

   An alert about Redis instances running out of disk space started the work; the outage itself showed up as backend errors about Redis connections right after the StatefulSet was recreated.

   Remediation

   • made the Redis masters forget obsolete node identities, reset the affected replicas and rejoined them, then waited for replication to catch up
   • identified the component that was actually running out of memory, scaled the backend service and raised memory limits for that component
   • temporarily relaxed probes so backend instances took traffic immediately instead of flapping between ready and not ready
   • reverted the temporary workarounds at 10:00 UTC

   Action items

   • Add startup probes so a Redis pod is not marked ready until its dataset is loaded
   • Build an own control plane for Redis that handles restarted pods stuck to an old master address
   • Reduce dependence on the single Redis cluster: move some important data to other stores, such as a separate Redis cluster per server, and use Kafka for some backend components

   Lessons

   The disk resize was not the problem; the cleanup afterwards was. An immutable field pushes operators toward deleting and recreating the controller, and that can set off a full reconcile. Startup and readiness probes are what slow a rolling restart down to a safe pace; without them a distributed store restarts in the most fragile way. Recovery plans must expect a cluster to heal slowly, and dependent services need to degrade gracefully while it does.

   View case
3. IL-0187 A brief Spring Cloud Config Server downtime failed every client application's liveness probe, because the actuator health check included the config server, and Kubernetes restarted the pods in a kill loop

   Yonghyun Kim2025

   Symptoms

   While the Spring Cloud Config Server was briefly down for an upgrade, every client application began to answer its liveness probe with a failure saying it could not reach the config server.

   Probe failuresContainer crash loopService outage

   Trigger

   A downtime upgrade of the Spring Cloud Config Server made it stop responding for a short time.

   Root cause

   Dependency failure

   Component

   Liveness and readiness probes

   Contributing factor

   Unreviewed defaultsAutomation amplificationExcessive blast radius

   Lessons

   A liveness probe that includes an external dependency turns a short outage of that dependency into a restart loop of every client. Keep external dependencies out of liveness checks and handle temporary problems with readiness probes. Libraries can add health indicators to Spring Boot's actuator health endpoint automatically, so check what the probe path actually reports. The author ran another Kubernetes environment for two years without liveness probes, relying on strict readiness probes and on the application terminating itself on unrecoverable faults.

   View case
4. IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiod

   Tetrate (unnamed U.S. federal customer)2025sev1

   Symptoms

   While the customer was updating the AMIs of its Amazon EKS nodes across more than 30 clusters running Istio, sidecar-injected workloads in one cluster failed to start and showed CrashLoopBackOff.

   Container crash loopProbe failures

   Trigger

   Updating the EKS node AMIs, which started hundreds of sidecar-injected pods at the same time.

   Root cause

   Dependency failure

   Component

   IstioAdmission webhookNode operating system image

   Contributing factor

   Unreviewed defaultsUnknown limit

   Detection

   Sidecar-injected workloads in the updated cluster failed to start and showed CrashLoopBackOff; the customer raised a Severity 1 ticket and a Tetrate engineer was reviewing logs within six minutes.

   Remediation

   • scaled istiod down temporarily from 40 replicas to 1 to slow Istio's internal processes and ease the load on the CA
   • deleted the CertificateSigningRequests stuck in the Approved, Failed state
   • restarted the pods gradually in a controlled order instead of all at once

   Lessons

   When a mesh hands certificate signing to an external CA, every pod start becomes a signing request to that CA, so a mass restart such as a node image rollout can overload it. Sidecars then abort under istiod's 10-second default CA timeout even though the CSRs are eventually signed. Restarting pods in controlled batches keeps the request rate within what the CA can handle. The author's point is that diagnosing such a failure means looking at the whole service mesh ecosystem, including its external dependencies, not only Istio.

   View case
5. IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layer

   OpenAI2025

   Symptoms

   From 3:30am to 7:08am PT, chatgpt.com and a small number of OpenAI APIs returned elevated error rates. The status page classed the incident as a full outage.

   High error rateContainer crash loopProbe failures

   Trigger

   A failure of Cosmos DB in ChatGPT, at OpenAI's service provider.

   Root cause

   Cloud provider outage

   Component

   Application database

   Contributing factor

   Health check tied to a dependency

   Remediation

   • the service provider recovered the affected Cosmos DB databases at 4:25am PT
   • fixed the web server's health checking logic, which ended the pod crash loop

   Action items

   • Change the web systems to degrade more gracefully when they fail, including concurrency control and better load shedding
   • Make the database layer more resilient to service provider outages (described as already underway)

   Lessons

   The database came back at 4:25am PT, yet users saw errors until 7:08am PT: the web tier's own health checks kept removing pods after the dependency had recovered. The planned changes target the web tier's behaviour under failure, with concurrency control and load shedding, rather than the database alone.

   View case

By root cause8