Cases2 cases

1 to 2 of 2

1. IL-0078 Enabling EKS secrets encryption with KMS through Terraform removed worker node IAM permissions and took down all ZenML Pro workspaces

   ZenML2025

   Symptoms

   Between 12 AM and 8 AM CEST on 11 May 2025, every ZenML Pro workspace server was unreachable, so customers could not use workspace UIs or pipelines. The ZenML Pro dashboard was not affected.

   Service outage

   Trigger

   At 23:38 on 10 May, secrets encryption with AWS KMS was enabled through Terraform as part of the EKS module upgrade.

   Root cause

   Software defect

   Component

   TerraformNode pool

   Contributing factor

   Unrepresentative stagingDeferred upgrades

   Detection

   Engineers found the cluster degraded when they resumed the upgrade at 06:00.

   Remediation

   • deleted and recreated the EKS node group with the correct IAM permissions and configuration
   • restored service access across all affected ZenML Pro environments

   Action items

   • Upgrade all Terraform modules (EKS, VPC) to current supported versions
   • Complete the EKS cluster upgrades as planned
   • Review the strategy for periodic Kubernetes and infrastructure upgrades to prevent a recurrence

   Lessons

   A successful upgrade on a much smaller staging cluster understated the risk of the same change in production. Skipping Kubernetes and module versions let upgrade complexity build up until one upgrade carried many changes at once.

   View case
2. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts

   DFDS2022

   Symptoms

   On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.

   Node NotReadyUpgrade stallIntermittent timeoutsIntermittent connection failures

   Trigger

   At 07:30 UTC on 2022-10-31 the daily node rollover replaced the existing nodes with nodes built from AMI v20221027, the latest AWS EKS node AMI, released on 2022-10-29 and later recalled because of a kernel issue.

   Root cause

   Software defect

   Component

   Node operating system imageLinux kernelkubeletAWS VPC CNINode launch templateTerraform

   Contributing factor

   No staged rolloutAutomation amplificationOverlapping failuresMonitoring blind spotNo health probes

   Detection

   The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.

   Remediation

   • cordoned, drained and deleted 15 stuck nodes one at a time to complete the vpc-cni upgrade
   • changed the Terraform infrastructure-modules so a specific AMI could be set
   • rolled the EC2 Auto Scaling group launch configuration back to the previous known good AMI through the pipeline

   Action items

   • Reach out to teams with suggestions about readiness and liveness probes
   • Create a dashboard for readiness and liveness probe coverage
   • Write a wiki article on readiness and liveness probes and their usefulness
   • Modify infrastructure-modules to allow specifying a particular AMI
   • Schedule a discussion on whether to take the latest AMI immediately and how to test new AMIs before they are deployed
   • Implement alerting on abnormal NodeNotReady events and on increased socket use
   • Create a learning path for Kubernetes within the team
   • Collect logs from nodes centrally
   • Create a Prometheus exporter that retrieves the latest available AMI, with a dashboard and alerts

   Lessons

   When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters. Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.

   View case

By root cause1