Cases13 cases

3 to 7 of 13

1. IL-0134 A power event at Lambda's AUS01 data center rebooted 246 GPU nodes, all spine switches and the VAST storage, failing customer workloads and leaving stale NFS mounts on VMs

   Lambda2026sev1

   Symptoms

   At 21:06 UTC on 14 February 2026 every GPU node in Lambda's AUS01 data center rebooted at the same moment, without warning or plan, affecting multiple customers. The nodes came back online quickly, but the storage they used had disconnected, and customer workloads failed.

   GPU unavailableStorage disconnected or stale mounts

   Trigger

   A power event at the AUS01 data center caused a loss of power.

   Root cause

   Dependency failure

   Component

   Data center power and coolingInfiniBand fabricShared network filesystem

   Contributing factor

   –

   Remediation

   • restored the compute fabric and InfiniBand connectivity, bringing all spine switches and most leaf switches back online
   • restored part of the UPS redundancy
   • powered up all nodes in AUS01, completed physical checks and started the storage back up
   • restored Starbase and the other storage-dependent services in us-west-3
   • told affected customers to reboot their VMs, or the hypervisor if needed, to clear stale or empty NFS mounts

   Lessons

   A power loss that reboots the network spine and the shared storage together with the GPU nodes takes longer to recover than the nodes themselves. The GPU nodes were back quickly, but the fabric was restored first, storage was started only after physical checks, and some VMs were left with stale NFS mounts that customers were told to fix with a VM reboot or, if that was not enough, a hypervisor reboot.

   View case
2. IL-0187 A brief Spring Cloud Config Server downtime failed every client application's liveness probe, because the actuator health check included the config server, and Kubernetes restarted the pods in a kill loop

   Yonghyun Kim2025

   Symptoms

   While the Spring Cloud Config Server was briefly down for an upgrade, every client application began to answer its liveness probe with a failure saying it could not reach the config server.

   Probe failuresContainer crash loopService outage

   Trigger

   A downtime upgrade of the Spring Cloud Config Server made it stop responding for a short time.

   Root cause

   Dependency failure

   Component

   Liveness and readiness probes

   Contributing factor

   Unreviewed defaultsAutomation amplificationExcessive blast radius

   Lessons

   A liveness probe that includes an external dependency turns a short outage of that dependency into a restart loop of every client. Keep external dependencies out of liveness checks and handle temporary problems with readiness probes. Libraries can add health indicators to Spring Boot's actuator health endpoint automatically, so check what the probe path actually reports. The author ran another Kubernetes environment for two years without liveness probes, relying on strict readiness probes and on the application terminating itself on unrecoverable faults.

   View case
3. IL-0162 LKE users could not reach their cluster control planes or the LKE Dashboard because an earlier database update made the internal LKE DNS service hang

   Linode2025

   Symptoms

   On 1 April 2025 users of Linode Kubernetes Engine (LKE) began having trouble connecting to their clusters' control planes, and the LKE Dashboard was affected as well. Services inside the clusters kept working. The first status update was posted at 20:28 UTC; at 21:33 UTC Linode said it was investigating an issue with DNS functionality within LKE.

   DNS resolution failuresControl plane unresponsive

   Trigger

   An earlier update to a database that the LKE DNS system depends on caused the DNS service to hang.

   Root cause

   Dependency failure

   Component

   Authoritative DNS servers and recordsApplication database

   Contributing factor

   –

   Detection

   The first status update, at 20:28 UTC on 1 April, said Linode was investigating and that users might have difficulty connecting to their cluster control plane. At 21:33 UTC Linode said the issue affected DNS functionality within LKE, and at 22:29 UTC that it had identified the underlying cause.

   Remediation

   • restarted the affected LKE DNS services

   Action items

   • Improve how the DNS services recover from transient connectivity failures to their dependencies
   • Improve monitoring to detect similar problems sooner
   • Update the deployment process to better coordinate changes between dependent systems

   Lessons

   A change to a database that the DNS service depends on can make the DNS service hang, which here cut off external access to the clusters and the LKE Dashboard while services inside the clusters kept working. The DNS service should recover by itself from transient failures of its dependencies, and changes to dependent systems need to be coordinated.

   View case
4. IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiod

   Tetrate (unnamed U.S. federal customer)2025sev1

   Symptoms

   While the customer was updating the AMIs of its Amazon EKS nodes across more than 30 clusters running Istio, sidecar-injected workloads in one cluster failed to start and showed CrashLoopBackOff.

   Container crash loopProbe failures

   Trigger

   Updating the EKS node AMIs, which started hundreds of sidecar-injected pods at the same time.

   Root cause

   Dependency failure

   Component

   IstioAdmission webhookNode operating system image

   Contributing factor

   Unreviewed defaultsUnknown limit

   Detection

   Sidecar-injected workloads in the updated cluster failed to start and showed CrashLoopBackOff; the customer raised a Severity 1 ticket and a Tetrate engineer was reviewing logs within six minutes.

   Remediation

   • scaled istiod down temporarily from 40 replicas to 1 to slow Istio's internal processes and ease the load on the CA
   • deleted the CertificateSigningRequests stuck in the Approved, Failed state
   • restarted the pods gradually in a controlled order instead of all at once

   Lessons

   When a mesh hands certificate signing to an external CA, every pod start becomes a signing request to that CA, so a mass restart such as a node image rollout can overload it. Sidecars then abort under istiod's 10-second default CA timeout even though the CSRs are eventually signed. Restarting pods in controlled batches keeps the request rate within what the CA can handle. The author's point is that diagnosing such a failure means looking at the whole service mesh ecosystem, including its external dependencies, not only Istio.

   View case
5. IL-0159 Docker Hub rate-limited Linode's IPv6 address space with 429 Too Many Requests, breaking image pulls for LKE and LKE-E clusters

   Linode2025

   Symptoms

   From about 21:38 UTC on 5 February 2025, image pulls from Docker Hub over IPv6 failed with 429 Too Many Requests errors across several Linode regions. Both unauthenticated and authenticated pull requests were refused.

   Image pull failures

   Trigger

   Docker Hub's abuse prevention measures started rate-limiting image pulls coming from Linode's IPv6 address space.

   Root cause

   Dependency failure

   Component

   Container registry and its credentialsVPC DNS resolver

   Contributing factor

   Unknown limit

   Detection

   The rate limiting began at about 21:38 UTC. The first status post, at 22:48 UTC, said Linode was aware of an issue with Docker Hub over IPv6 causing rate limiting and was investigating, and at 01:06 UTC on 6 February the status page said the issue had been identified as rate limiting of image pull requests to Docker Hub over IPv6.

   Remediation

   • changed the DNS resolvers to return an A record instead of an AAAA record so image pulls to Docker Hub used IPv4

   Action items

   • Find long-term solutions so that Docker Hub's abuse prevention measures do not affect Linode services

   Lessons

   A registry's abuse limit that groups IPv6 addresses by their first 64 bits can block a provider's whole address range at once, and authentication does not help. Answering DNS lookups with IPv4 addresses only was the quick way to move image pulls off the blocked range.

   View case

By root cause1