Cases2 cases

1 to 2 of 2

1. IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiod

   Tetrate (unnamed U.S. federal customer)2025sev1

   Symptoms

   While the customer was updating the AMIs of its Amazon EKS nodes across more than 30 clusters running Istio, sidecar-injected workloads in one cluster failed to start and showed CrashLoopBackOff.

   Container crash loopProbe failures

   Trigger

   Updating the EKS node AMIs, which started hundreds of sidecar-injected pods at the same time.

   Root cause

   Dependency failure

   Component

   IstioAdmission webhookNode operating system image

   Contributing factor

   Unreviewed defaultsUnknown limit

   Detection

   Sidecar-injected workloads in the updated cluster failed to start and showed CrashLoopBackOff; the customer raised a Severity 1 ticket and a Tetrate engineer was reviewing logs within six minutes.

   Remediation

   • scaled istiod down temporarily from 40 replicas to 1 to slow Istio's internal processes and ease the load on the CA
   • deleted the CertificateSigningRequests stuck in the Approved, Failed state
   • restarted the pods gradually in a controlled order instead of all at once

   Lessons

   When a mesh hands certificate signing to an external CA, every pod start becomes a signing request to that CA, so a mass restart such as a node image rollout can overload it. Sidecars then abort under istiod's 10-second default CA timeout even though the CSRs are eventually signed. Restarting pods in controlled batches keeps the request rate within what the CA can handle. The author's point is that diagnosing such a failure means looking at the whole service mesh ecosystem, including its external dependencies, not only Istio.

   View case
2. IL-0159 Docker Hub rate-limited Linode's IPv6 address space with 429 Too Many Requests, breaking image pulls for LKE and LKE-E clusters

   Linode2025

   Symptoms

   From about 21:38 UTC on 5 February 2025, image pulls from Docker Hub over IPv6 failed with 429 Too Many Requests errors across several Linode regions. Both unauthenticated and authenticated pull requests were refused.

   Image pull failures

   Trigger

   Docker Hub's abuse prevention measures started rate-limiting image pulls coming from Linode's IPv6 address space.

   Root cause

   Dependency failure

   Component

   Container registry and its credentialsVPC DNS resolver

   Contributing factor

   Unknown limit

   Detection

   The rate limiting began at about 21:38 UTC. The first status post, at 22:48 UTC, said Linode was aware of an issue with Docker Hub over IPv6 causing rate limiting and was investigating, and at 01:06 UTC on 6 February the status page said the issue had been identified as rate limiting of image pull requests to Docker Hub over IPv6.

   Remediation

   • changed the DNS resolvers to return an A record instead of an AAAA record so image pulls to Docker Hub used IPv4

   Action items

   • Find long-term solutions so that Docker Hub's abuse prevention measures do not affect Linode services

   Lessons

   A registry's abuse limit that groups IPv6 addresses by their first 64 bits can block a provider's whole address range at once, and authentication does not help. Answering DNS lookups with IPv4 addresses only was the quick way to move image pulls off the blocked range.

   View case

By root cause1