Cases2 cases

1 to 2 of 2

1. IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiod

   Tetrate (unnamed U.S. federal customer)2025sev1

   Symptoms

   While the customer was updating the AMIs of its Amazon EKS nodes across more than 30 clusters running Istio, sidecar-injected workloads in one cluster failed to start and showed CrashLoopBackOff.

   Container crash loopProbe failures

   Trigger

   Updating the EKS node AMIs, which started hundreds of sidecar-injected pods at the same time.

   Root cause

   Dependency failure

   Component

   IstioAdmission webhookNode operating system image

   Contributing factor

   Unreviewed defaultsUnknown limit

   Detection

   Sidecar-injected workloads in the updated cluster failed to start and showed CrashLoopBackOff; the customer raised a Severity 1 ticket and a Tetrate engineer was reviewing logs within six minutes.

   Remediation

   • scaled istiod down temporarily from 40 replicas to 1 to slow Istio's internal processes and ease the load on the CA
   • deleted the CertificateSigningRequests stuck in the Approved, Failed state
   • restarted the pods gradually in a controlled order instead of all at once

   Lessons

   When a mesh hands certificate signing to an external CA, every pod start becomes a signing request to that CA, so a mass restart such as a node image rollout can overload it. Sidecars then abort under istiod's 10-second default CA timeout even though the CSRs are eventually signed. Restarting pods in controlled batches keeps the request rate within what the CA can handle. The author's point is that diagnosing such a failure means looking at the whole service mesh ecosystem, including its external dependencies, not only Istio.

   View case
2. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes

   DFDS2022

   Symptoms

   At 11:38 UTC on 3 August 2022 an end-user team told the platform team that applying a Service resource to one of the Kubernetes clusters failed.

   Deploys failingContainer crash loop

   Trigger

   A certificate chain in a third-party application expired, through no action of DFDS, and the outdated Kyverno installation got stuck in a crash loop.

   Root cause

   Certificate or credential expiry

   Component

   Policy engineAdmission webhook

   Contributing factor

   Deferred upgradesUnreviewed defaultsMonitoring blind spotIgnored early signal

   Detection

   An end-user team alerted the platform team at 11:38 UTC that applying a Service resource failed; the platform team started looking into it at 11:50 UTC.

   Remediation

   • reproduced the issue in a sandbox cluster and deployed the fix there first
   • updated Kyverno to a newer Helm chart version in the affected production cluster

   Action items

   • Implement monitoring on Kyverno
   • Create internal documentation on the Kyverno setup
   • Investigate the exact certificate issue in the third-party application
   • Consider changing the failurePolicy on ClusterPolicy resources going forward

   Lessons

   Operational issues should be added to the 'Ops Day to Day' board from the start.

   View case

By root cause2