IL-0078 ZenML · 2025-05-11kubernetesaws
The production EKS cluster that hosts ZenML Pro workspaces was on an outdated Kubernetes version and AMI type. On 9 May 2025 the staging cluster was fully upgraded with little disruption, which also showed that the EKS and VPC Terraform modules had to be upgraded to finish the cluster upgrade. The production upgrade was approved for the weekend of 10 to 11 May.
The upgrade started at 18:00 on 10 May while most workspaces were idle, and the VPC and EKS module upgrades went through cleanly. At 23:38 KMS secrets encryption was enabled through Terraform as part of the EKS module upgrade; it took much longer than on staging, likely because of the number of nodes and secrets. From 00:45 the upgrade was left to run overnight, since no problems were expected.
Applying the encryption through Terraform hit an undocumented bug: Terraform also removed key IAM permissions from the worker nodes, which made the workspace server pods unreachable from outside the cluster. Editing the IAM roles did not restore access; the node group had to be deleted and recreated with the correct IAM permissions and configuration, and service was fully restored around 07:30. The report counts about eight hours of downtime for all production workspaces, with no data loss, corruption or unauthorised access.
The report names three contributing factors: outdated EKS and VPC Terraform modules and legacy AMI versions, upgrade complexity that built up because Kubernetes and module versions had been skipped, and a risk judgement based on a staging cluster of much smaller scale.
Between 12 AM and 8 AM CEST on 11 May 2025, every ZenML Pro workspace server was unreachable, so customers could not use workspace UIs or pipelines. The ZenML Pro dashboard was not affected.
When engineers resumed an overnight cluster upgrade at 06:00, they found the production EKS cluster degraded. The step that enabled secrets encryption with AWS KMS had succeeded, but Terraform had also removed key IAM permissions from the EKS worker nodes, and the ZenML Pro server pods could no longer be reached from outside the cluster.
Changing the IAM roles to restore access did not work. Between 06:00 and 07:30 the outage continued while the team troubleshot it.
Symptom Service outage
Root cause Software defect
Engineers found the cluster degraded when they resumed the upgrade at 06:00.
A successful upgrade on a much smaller staging cluster understated the risk of the same change in production. Skipping Kubernetes and module versions let upgrade complexity build up until one upgrade carried many changes at once.
IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binariesMicrosoft · 2025same root cause · shared Node pool
IL-0161 A faulty earlier API update made new LKE cluster provisioning in Toronto time out for three days, leaving clusters with no node pools or the wrong number of nodesLinode · 2024same root cause · shared Node pool
IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022same root cause · shared Terraform
IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutesGrafana Labs · 2020same root cause · shared Node pool
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.