Incident Library
Applied filters Root causeSoftware defectComponentTerraformClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0078 Enabling EKS secrets encryption with KMS through Terraform removed worker node IAM permissions and took down all ZenML Pro workspaces
    ZenML2025
    Symptoms

    Between 12 AM and 8 AM CEST on 11 May 2025, every ZenML Pro workspace server was unreachable, so customers could not use workspace UIs or pipelines. The ZenML Pro dashboard was not affected.

    Service outage
    Trigger

    At 23:38 on 10 May, secrets encryption with AWS KMS was enabled through Terraform as part of the EKS module upgrade.

    Root cause
    Software defect
    Component
    TerraformNode pool
    Contributing factor
    Unrepresentative stagingDeferred upgrades
    Detection

    Engineers found the cluster degraded when they resumed the upgrade at 06:00.

    Remediation
    • deleted and recreated the EKS node group with the correct IAM permissions and configuration
    • restored service access across all affected ZenML Pro environments
    Action items
    • Upgrade all Terraform modules (EKS, VPC) to current supported versions
    • Complete the EKS cluster upgrades as planned
    • Review the strategy for periodic Kubernetes and infrastructure upgrades to prevent a recurrence
    Lessons

    A successful upgrade on a much smaller staging cluster understated the risk of the same change in production. Skipping Kubernetes and module versions let upgrade complexity build up until one upgrade carried many changes at once.

    View case
  2. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts
    DFDS2022
    Symptoms

    On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.

    Node NotReadyUpgrade stallIntermittent timeoutsIntermittent connection failures
    Trigger

    At 07:30 UTC on 2022-10-31 the daily node rollover replaced the existing nodes with nodes built from AMI v20221027, the latest AWS EKS node AMI, released on 2022-10-29 and later recalled because of a kernel issue.

    Root cause
    Software defect
    Component
    Node operating system imageLinux kernelkubeletAWS VPC CNINode launch templateTerraform
    Contributing factor
    No staged rolloutAutomation amplificationOverlapping failuresMonitoring blind spotNo health probes
    Detection

    The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.

    Remediation
    • cordoned, drained and deleted 15 stuck nodes one at a time to complete the vpc-cni upgrade
    • changed the Terraform infrastructure-modules so a specific AMI could be set
    • rolled the EC2 Auto Scaling group launch configuration back to the previous known good AMI through the pipeline
    Action items
    • Reach out to teams with suggestions about readiness and liveness probes
    • Create a dashboard for readiness and liveness probe coverage
    • Write a wiki article on readiness and liveness probes and their usefulness
    • Modify infrastructure-modules to allow specifying a particular AMI
    • Schedule a discussion on whether to take the latest AMI immediately and how to test new AMIs before they are deployed
    • Implement alerting on abnormal NodeNotReady events and on increased socket use
    • Create a learning path for Kubernetes within the team
    • Collect logs from nodes centrally
    • Create a Prometheus exporter that retrieves the latest available AMI, with a dashboard and alerts
    Lessons

    When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters. Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.

    View case
Check your cluster for this

These 2 cases involve Terraform and kubelet. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect2 cases

Component
Terraform2kubelet1Node launch template1Linux kernel1Node operating system image1Node pool1AWS VPC CNI1
Contributing factor
Automation amplification1Monitoring blind spot1No health probes1No staged rollout1Overlapping failures1Deferred upgrades1Unrepresentative staging1
Organisation
DFDS1ZenML1
  1. IL-0078 Enabling EKS secrets encryption with KMS through Terraform removed worker node IAM permissions and took down all ZenML Pro workspaces
    ZenML2025
  2. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts
    DFDS2022