Cases13 cases

6 to 10 of 13

1. IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiod

   Tetrate (unnamed U.S. federal customer)2025sev1

   Symptoms

   While the customer was updating the AMIs of its Amazon EKS nodes across more than 30 clusters running Istio, sidecar-injected workloads in one cluster failed to start and showed CrashLoopBackOff.

   Container crash loopProbe failures

   Trigger

   Updating the EKS node AMIs, which started hundreds of sidecar-injected pods at the same time.

   Root cause

   Dependency failure

   Component

   IstioAdmission webhookNode operating system image

   Contributing factor

   Unreviewed defaultsUnknown limit

   Detection

   Sidecar-injected workloads in the updated cluster failed to start and showed CrashLoopBackOff; the customer raised a Severity 1 ticket and a Tetrate engineer was reviewing logs within six minutes.

   Remediation

   • scaled istiod down temporarily from 40 replicas to 1 to slow Istio's internal processes and ease the load on the CA
   • deleted the CertificateSigningRequests stuck in the Approved, Failed state
   • restarted the pods gradually in a controlled order instead of all at once

   Lessons

   When a mesh hands certificate signing to an external CA, every pod start becomes a signing request to that CA, so a mass restart such as a node image rollout can overload it. Sidecars then abort under istiod's 10-second default CA timeout even though the CSRs are eventually signed. Restarting pods in controlled batches keeps the request rate within what the CA can handle. The author's point is that diagnosing such a failure means looking at the whole service mesh ecosystem, including its external dependencies, not only Istio.

   View case
2. IL-0159 Docker Hub rate-limited Linode's IPv6 address space with 429 Too Many Requests, breaking image pulls for LKE and LKE-E clusters

   Linode2025

   Symptoms

   From about 21:38 UTC on 5 February 2025, image pulls from Docker Hub over IPv6 failed with 429 Too Many Requests errors across several Linode regions. Both unauthenticated and authenticated pull requests were refused.

   Image pull failures

   Trigger

   Docker Hub's abuse prevention measures started rate-limiting image pulls coming from Linode's IPv6 address space.

   Root cause

   Dependency failure

   Component

   Container registry and its credentialsVPC DNS resolver

   Contributing factor

   Unknown limit

   Detection

   The rate limiting began at about 21:38 UTC. The first status post, at 22:48 UTC, said Linode was aware of an issue with Docker Hub over IPv6 causing rate limiting and was investigating, and at 01:06 UTC on 6 February the status page said the issue had been identified as rate limiting of image pull requests to Docker Hub over IPv6.

   Remediation

   • changed the DNS resolvers to return an A record instead of an AAAA record so image pulls to Docker Hub used IPv4

   Action items

   • Find long-term solutions so that Docker Hub's abuse prevention measures do not affect Linode services

   Lessons

   A registry's abuse limit that groups IPv6 addresses by their first 64 bits can block a provider's whole address range at once, and authentication does not help. Answering DNS lookups with IPv4 addresses only was the quick way to move image pulls off the blocked range.

   View case
3. IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training cluster

   Imbue2024

   Symptoms

   When Imbue brought up a cluster of 4,088 H100 GPUs on 511 hosts, about 10% of the machines failed to boot, mostly from physical problems: unconnected or miswired Ethernet cables, iDRAC hardware issues, broken power supplies, bad NVMe drives, missing internal wires, and network cards or GPUs that did not show up. In single-node GPU tests, Ubuntu server logs showed links between the GPUs and the PCIe buses or network cards reporting "limited width: x4 < x16". A few hosts had NVMe drives that locked up the whole machine when touched, drives that appeared in random order and got the OS installed on the wrong one, wrong temperature readings that kept fans at 100%, and CPUs limited to 2 GHz.

   Link flappingNCCL timeoutThroughput degradationGPU unavailableTraining job failure

   Trigger

   Bringing a new InfiniBand-connected H100 cluster into service and running the first multi-node training jobs on it.

   Root cause

   Dependency failure

   Component

   InfiniBand fabricNCCLGPU driverPyTorch

   Contributing factor

   Unverified assumptionMonitoring blind spotPartial rollout

   Detection

   Automated hardware checks during provisioning, UFM alerts on temperature, error rates and flapping ports, Prometheus node exporter and DCGM metrics, per-batch MFU monitoring, and live debugging of stalled processes with Py-Spy, GDB and NVIDIA Nsight.

   Remediation

   • Automated checks for broken machines and sent faulty ones back to Dell or to data center staff
   • Reseated GPUs and internal PCIe cables and applied firmware upgrades
   • Downgraded the NVIDIA driver that caused wrong temperature readings
   • Rewired the top tier of the InfiniBand fabric into a single network and added checks that the wiring matched the design
   • Cleaned and reseated alerting ports and disabled the remaining alerting transceivers
   • Ran a burn-in workload through every InfiniBand port and disabled the ports that did not survive it
   • Enabled GPUDirect RDMA with an extra kernel module and PCIe ACS disabled
   • Disabled automatic garbage collection and scheduled it at set intervals on all hosts

   Action items

   • Release host-level health checks that confirm a host is free from known errors
   • Release an NCCL patch that improves logging around errors and stalls
   • Release a GPU stress test and NVLink and InfiniBand networking tests
   • Release a script that parses the UFM event log and decides which network ports to disable
   • Release a script that generates a burn-in workload exercising every InfiniBand link
   • Relaunch crashed runs automatically from the last checkpoint on the machines that pass health checks
   • Replace the disabled alerting transceivers

   Lessons

   Keep 10 to 20% more machines than a run needs, and network them so any working subset can be used. On the newest hardware expect about 3% of machines to break each week, mostly the same few machines breaking in different ways, so growing a set of known reliable "golden" hosts works better than chasing random failures. Write tests and automation for every kind of failure met, because each one comes back, and write tools that make opaque errors readable. Change only one thing at a time, and double-check the claims of new tools and people before building on them.

   View case
4. IL-0021 GKE GPU nodes on Container-Optimized OS could not download NVIDIA drivers (403 Forbidden) and came up without working GPUs

   Google Cloud2024

   Symptoms

   From 08:00 US/Pacific on 12 March 2024, GKE GPU nodes running Container-Optimized OS (COS) that were newly created or recreated came up with GPU drivers that did not work, so workloads that used the GPUs could not run. GKE showed elevated errors.

   GPU unavailableNode NotReady

   Root cause

   Dependency failure

   Component

   GPU driverNode operating system imageNode pool

   Contributing factor

   Automation amplification

   Detection

   The report does not say how Google detected the issue. Affected users saw "Failed to download GPU driver installer, status: 403 Forbidden" on the GPU node when installing the driver.

   Remediation

   • Halted automatic node recreations, which attempt GPU driver downloads, until the issue was mitigated
   • Worked with the driver partner to resolve access to the driver download bucket

   Lessons

   The status page report states no lessons or follow-up actions.

   View case
5. IL-0135 A power source switch during UPS maintenance cut power to a row of GPU servers and shut down Crusoe Cloud VMs in us-northcentral1

   Crusoe Cloud2023

   Symptoms

   At 20:31 UTC on 15 November 2023, some GPU virtual machines in us-northcentral1-a became unavailable. One row of servers in a Crusoe Cloud data center in the us-northcentral1 region had lost power, and every server in the row powered off together with the VMs running on it.

   GPU unavailableService outage

   Trigger

   While the row's UPS was in bypass mode for routine maintenance, power switched from the primary source to a backup source for reasons unrelated to the maintenance.

   Root cause

   Dependency failure

   Component

   Data center power and cooling

   Contributing factor

   Single point of failureOverlapping failures

   Detection

   The power failure happened at 20:31 UTC. The first status post, at 21:07 UTC, said certain VMs in us-northcentral1-a were unavailable, and at 21:31 UTC the status page said the affected VMs had been identified.

   Remediation

   • identified the VMs on the row that lost power
   • restarted the affected VMs after power was restored

   Action items

   • Review the standard operating procedures for UPS maintenance to minimise customer impact during future maintenance
   • Build future data centers with redundant UPSs
   • Improve recovery procedures to reduce mean time to resolution

   Lessons

   When the only UPS for a row is in bypass for maintenance, the row is exposed to any power event during that window, even one unrelated to the work. Crusoe Cloud's response is redundant UPSs in future data centers, a review of maintenance procedures, and faster recovery of VMs after power returns.

   View case

By root cause1