Cases18 cases

3 to 7 of 18

1. IL-0187 A brief Spring Cloud Config Server downtime failed every client application's liveness probe, because the actuator health check included the config server, and Kubernetes restarted the pods in a kill loop

   Yonghyun Kim2025

   Symptoms

   While the Spring Cloud Config Server was briefly down for an upgrade, every client application began to answer its liveness probe with a failure saying it could not reach the config server.

   Probe failuresContainer crash loopService outage

   Trigger

   A downtime upgrade of the Spring Cloud Config Server made it stop responding for a short time.

   Root cause

   Dependency failure

   Component

   Liveness and readiness probes

   Contributing factor

   Unreviewed defaultsAutomation amplificationExcessive blast radius

   Lessons

   A liveness probe that includes an external dependency turns a short outage of that dependency into a restart loop of every client. Keep external dependencies out of liveness checks and handle temporary problems with readiness probes. Libraries can add health indicators to Spring Boot's actuator health endpoint automatically, so check what the probe path actually reports. The author ran another Kubernetes environment for two years without liveness probes, relying on strict readiness probes and on the application terminating itself on unrecoverable faults.

   View case
2. IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiod

   Tetrate (unnamed U.S. federal customer)2025sev1

   Symptoms

   While the customer was updating the AMIs of its Amazon EKS nodes across more than 30 clusters running Istio, sidecar-injected workloads in one cluster failed to start and showed CrashLoopBackOff.

   Container crash loopProbe failures

   Trigger

   Updating the EKS node AMIs, which started hundreds of sidecar-injected pods at the same time.

   Root cause

   Dependency failure

   Component

   IstioAdmission webhookNode operating system image

   Contributing factor

   Unreviewed defaultsUnknown limit

   Detection

   Sidecar-injected workloads in the updated cluster failed to start and showed CrashLoopBackOff; the customer raised a Severity 1 ticket and a Tetrate engineer was reviewing logs within six minutes.

   Remediation

   • scaled istiod down temporarily from 40 replicas to 1 to slow Istio's internal processes and ease the load on the CA
   • deleted the CertificateSigningRequests stuck in the Approved, Failed state
   • restarted the pods gradually in a controlled order instead of all at once

   Lessons

   When a mesh hands certificate signing to an external CA, every pod start becomes a signing request to that CA, so a mass restart such as a node image rollout can overload it. Sidecars then abort under istiod's 10-second default CA timeout even though the CSRs are eventually signed. Restarting pods in controlled batches keeps the request rate within what the CA can handle. The author's point is that diagnosing such a failure means looking at the whole service mesh ecosystem, including its external dependencies, not only Istio.

   View case
3. IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layer

   OpenAI2025

   Symptoms

   From 3:30am to 7:08am PT, chatgpt.com and a small number of OpenAI APIs returned elevated error rates. The status page classed the incident as a full outage.

   High error rateContainer crash loopProbe failures

   Trigger

   A failure of Cosmos DB in ChatGPT, at OpenAI's service provider.

   Root cause

   Cloud provider outage

   Component

   Database

   Contributing factor

   Health check tied to a dependency

   Remediation

   • the service provider recovered the affected Cosmos DB databases at 4:25am PT
   • fixed the web server's health checking logic, which ended the pod crash loop

   Action items

   • Change the web systems to degrade more gracefully when they fail, including concurrency control and better load shedding
   • Make the database layer more resilient to service provider outages (described as already underway)

   Lessons

   The database came back at 4:25am PT, yet users saw errors until 7:08am PT: the web tier's own health checks kept removing pods after the dependency had recovered. The planned changes target the web tier's behaviour under failure, with concurrency control and load shedding, rather than the database alone.

   View case
4. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one

   Wikimedia Foundation2024

   Symptoms

   At 05:01 UTC on 9 October 2024 a page fired for GatewayBackendErrorsHigh, elevated 5xx errors from page-analytics_cluster in codfw, followed by a growing number of KubernetesCalicoDown alerts for WikiKube nodes not running the calico-node pod and by MediaWikiMemcachedHighErrorRate. For about an hour uncached MediaWiki requests, such as those of logged-in users, edits and API calls, saw high latency or errors.

   High error rateHigh latencyContainer crash loopProbe failuresControl plane lag

   Trigger

   An operator used the new mwscript-k8s utility to run a batch of video transcode maintenance, the TimedMediaHandler requeueTranscodes.php script. mwscript-k8s deploys a whole helm release for every script invocation, so the batch created close to 2k Calico NetworkPolicies along with many other resources over several hours.

   Root cause

   Capacity exhaustion

   Component

   CalicoHelmJobs and CronJobs

   Contributing factor

   Change with a delayed effectNo runbook

   Detection

   Automated monitoring detected the issue and paged the on-call SRE: GatewayBackendErrorsHigh at 05:01:36 UTC, 16 KubernetesCalicoDown alerts at 05:03:32 UTC and MediaWikiMemcachedHighErrorRate at 05:04:15 UTC.

   Remediation

   • Raised the resource limits of the Calico and Typha pods and applied the change with helmfile, which redeployed all Calico pods

   Action items

   • Stop mwscript-k8s from creating so many resources

   Lessons

   The incident came close to a full outage. Automated monitoring caught it and many responders found the cause, but end users were affected and responders were unsure whether they could switch over to eqiad. The problem was noticed just before the hand-off from Americas to EMEA SRE shifts, a period with a long-known lack of coverage, and it helped that the transcodes were not queued on a Friday evening in the US.

   View case
5. IL-0495 HubSpot's internal load balancers on Kubernetes were falsely marked unhealthy and the rest collapsed under the traffic

   HubSpot2024

   Symptoms

   Between 14:20 and 15:16 UTC on 27 August 2024, some HubSpot customers could not log in to their accounts, reach their content or use HubSpot tools. Requests were slow or failed because HubSpot's internal load balancers, which run on Kubernetes and spread traffic across the backend services, were overloaded.

   Probe failuresHigh latencyHigh error rate

   Trigger

   Routine operation of a critical component of HubSpot's traffic management system, from about 14:08 UTC, ran path handling updates in the load balancer stack, and an issue in that stack made those updates block health check responses.

   Root cause

   Software defect

   Component

   Ingress controllerLiveness and readiness probes

   Contributing factor

   Automation amplificationNo headroom

   Detection

   Monitoring detected widespread performance degradation across the platform at 14:23 UTC and the incident response team was mobilised. HubSpot says it had identified the issue internally by 14:20 UTC. The status page was not updated until 14:44 UTC, which left customers with outdated information.

   Remediation

   • Tried to increase the capacity of the remaining load balancer instances
   • Brought new load balancer instances online, which relieved the pressure and let traffic return to normal

   Action items

   • Change the health check configuration so health checks do not time out falsely
   • Redesign the health check system so health checks get priority and keep running during operations such as configuration updates
   • Increase the capacity of the load balancers for sudden traffic spikes
   • Change the load balancing architecture to detect traffic surges earlier and distribute load dynamically in real time
   • Update the status page automatically for large-scale outages
   • Integrate the monitoring systems more tightly with the communication tools
   • Expand load testing to simulate high-traffic scenarios
   • Introduce chaos engineering to test platform resilience regularly
   • Add traffic pattern metrics, including the ratio of current connections to the maximum allowed
   • Add auto-scaling that adjusts resource capacity to traffic patterns

   Lessons

   HubSpot concluded that earlier detection of the abnormal traffic surge could have prevented the cascading failure, and that health checks have to keep running during operations such as configuration updates so that working load balancers are not removed together. The late status page update caused confusion for customers.

   View case

By root cause8