Cases4 cases

1 to 4 of 4

1. IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiod

   Tetrate (unnamed U.S. federal customer)2025sev1

   Symptoms

   While the customer was updating the AMIs of its Amazon EKS nodes across more than 30 clusters running Istio, sidecar-injected workloads in one cluster failed to start and showed CrashLoopBackOff.

   Container crash loopProbe failures

   Trigger

   Updating the EKS node AMIs, which started hundreds of sidecar-injected pods at the same time.

   Root cause

   Dependency failure

   Component

   IstioAdmission webhookNode operating system image

   Contributing factor

   Unreviewed defaultsUnknown limit

   Detection

   Sidecar-injected workloads in the updated cluster failed to start and showed CrashLoopBackOff; the customer raised a Severity 1 ticket and a Tetrate engineer was reviewing logs within six minutes.

   Remediation

   • scaled istiod down temporarily from 40 replicas to 1 to slow Istio's internal processes and ease the load on the CA
   • deleted the CertificateSigningRequests stuck in the Approved, Failed state
   • restarted the pods gradually in a controlled order instead of all at once

   Lessons

   When a mesh hands certificate signing to an external CA, every pod start becomes a signing request to that CA, so a mass restart such as a node image rollout can overload it. Sidecars then abort under istiod's 10-second default CA timeout even though the CSRs are eventually signed. Restarting pods in controlled batches keeps the request rate within what the CA can handle. The author's point is that diagnosing such a failure means looking at the whole service mesh ecosystem, including its external dependencies, not only Istio.

   View case
2. IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networking

   DigitalOcean2023

   Symptoms

   Customers of DigitalOcean Managed Kubernetes (DOKS) saw worker nodes go NotReady between 06:00 and 07:00 UTC and stay unavailable, taking down the services on them, until the nodes were replaced or rebooted; some nodes needed several replacements or reboots. The first sporadic reports on October 8 said replacing the node fixed it. The same thing happened again around 06:00 UTC on October 9, more widely and across many regions, and again on October 10. Only clusters on versions 1.27.6-do.0 and 1.28.2-do.0 were affected.

   Node NotReadyService outage

   Trigger

   A systemd timer in the DigitalOcean Debian image started apt upgrade daily at 06:00 UTC, with a random delay of up to 60 minutes.

   Root cause

   Configuration change

   Component

   Node operating system imageLinux kernelkubelet

   Contributing factor

   Unreviewed defaultsLimited debug accessChange with a delayed effect

   Detection

   Sporadic customer reports of failed worker nodes arrived at 07:43 UTC on October 8; internal incident response started at 07:10 UTC on October 9 after broader impact across regions.

   Remediation

   • Advised customers to replace or reboot affected worker nodes
   • Arranged SSH and console access to worker nodes while they were healthy, and live-debugged an affected node on October 10
   • Released a DaemonSet to all affected clusters that disables the apt upgrade systemd timer on every worker node
   • Released new node images without the auto-update process

   Action items

   • Disable the timer during node provisioning, then remove the DaemonSet from all clusters
   • Add a check to the internal conformance test suite for new cluster versions that the auto-upgrade process and other unwanted timers stay disabled
   • Support logging into worker nodes even when normal access paths are broken

   Lessons

   An operating system auto-update timer left enabled in a node image can change the kernel and systemd underneath Kubernetes, and a timeout that kills the upgrade half-way can leave a node without networking. Failures that start at the same time every day point to a scheduled job on the nodes, and systemd timers belong on that list next to cron jobs. When the failure takes out a node's network, an investigation that depends on logging into the node stalls; access paths that work while the node is broken shorten diagnosis.

   View case
3. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers

   Datadog2023

   Symptoms

   Between 06:00 and 07:00 UTC on 8 March 2023, Kubernetes nodes in several Datadog regions lost network connectivity at the same time, on AWS, Google Cloud and Azure. All services across multiple regions were affected, including the regions that host Datadog's CI/CD and automation tools. Datadog's regions are not connected to each other and changes are rolled out one region at a time, so a simultaneous failure on several cloud providers looked implausible at first.

   Node NotReadyService outageThroughput degradation

   Trigger

   On 7 March 2023 Ubuntu published a systemd patch for a CVE (249.11-0ubuntu3.7). Ubuntu's default unattended upgrades installed it on the nodes between 06:00 and 07:00 UTC on 8 March, and installing it restarted systemd-networkd.

   Root cause

   Breaking change in an upgrade

   Component

   systemd-networkdCiliumNode operating system image

   Contributing factor

   Unreviewed defaultsChange with a delayed effectNo staged rolloutAutomation amplification

   Remediation

   • restarted thousands of disconnected instances on Google Cloud and Azure through the cloud APIs, keeping their local disks
   • rebuilt the data lost with terminated AWS instances' local disks through replication, reprocessing and restores from backup

   Lessons

   A change in a base operating system component can stay dormant for months until something restarts that component. Here the trigger was a security patch, and the change reached every region at once because automatic OS updates ran outside the progressive, region-by-region process used for every other change. Cloud providers react differently to an instance that loses its network. Google Cloud and Azure left the instances running, so a restart recovered them with their disks; AWS replaced them quickly, which brought stateless services back fast but destroyed the local-disk data the stateful services needed.

   View case
4. IL-0057 Higher p99 latency after moving a reverse proxy to EKS, from outgoing connections delayed by a SNAT conntrack race

   Prezi2019

   Symptoms

   Prometheus alerts fired on tail latency: from 7/24, the p99 response time of a few services went up after Prezi moved its Nginx-based reverse proxy, smartgateway, from dedicated Elastic Beanstalk machines onto shared Kubernetes worker nodes. The bump affected only the tail, so the earlier monitoring of average response times would most certainly have missed it.

   High latencyIntermittent timeouts

   Trigger

   Moving the reverse proxy onto shared EKS worker nodes, where its outgoing connections went through source NAT on the node and some SYN packets were dropped by a race in Netfilter's conntrack.

   Root cause

   DNS or network fault

   Component

   conntrack tableAWS VPC CNILinux kernelNode operating system image

   Contributing factor

   Unreviewed defaultsUnpatched known bug

   Detection

   Prometheus alerts on p99 latency fired after the reverse proxy migration.

   Remediation

   • added AWS X-Ray tracing and an $upstream_connect_time trace attribute to locate the delay
   • ran more pods of the service on more nodes, which reduced but did not end the delays
   • set AWS_VPC_K8S_CNI_EXTERNALSNAT=true on the AWS VPC CNI to turn off SNAT on the nodes

   Lessons

   Alerting on tail latency instead of averages caught a problem that would otherwise have gone unnoticed, and tracing turned a vague slowdown into a measured connect delay. Delays of exactly 1 or 3 seconds on new outgoing connections point to dropped SYN packets, and on EKS the SNAT done by the VPC CNI is where to look.

   View case

By root cause4