Cases4 cases

1 to 4 of 4

1. IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binaries

   Microsoft2025

   Symptoms

   Creating or scaling a node pool on AKS 1.31.8 failed: the VMSS instances never finished provisioning. This started after planned maintenance moved the node image from 202507.* to 202508.*, in a cluster where nodes must send all outbound traffic through an internal HTTP/HTTPS proxy and direct egress is blocked. It happened with both Ubuntu and AzureLinux images.

   Scale-up failureUpgrade stall

   Trigger

   A node image auto-upgrade from 202507.* to 202508.* during planned maintenance, on clusters whose Kubernetes patch version was not pre-cached in the new image.

   Root cause

   Software defect

   Component

   Node operating system imageNode pool

   Contributing factor

   Cache-masked failureAutomation amplification

   Detection

   The customer saw VMSS instance provisioning fail with a vmssCSE error in the VMSS activity logs and reported it as a GitHub issue.

   Remediation

   • The customer pinned the node image to a 202507.* snapshot on clusters running 1.31.8
   • The customer upgraded clusters to AKS 1.31.9 or 1.31.10, whose binaries are cached in the 202508.* images
   • The AKS team hot-patched VHD 20250811 so the proxy configuration is used for uncached binary downloads and made it globally available in public regions on 2025-09-04; it also announced the hotfix for VHD 20250820
   • The AKS team told users to reimage their nodes with az aks nodepool upgrade --node-image-only to pick up the hotfixed image

   Action items

   • Apply the proxy fix in all future VHDs

   Lessons

   A bootstrap path that runs only when an artefact is missing from a cache is easy to break without noticing: here only clusters on Kubernetes versions not pre-cached in the node image ever downloaded binaries, so only they met the proxy regression. Testing node provisioning behind a mandatory egress proxy, on older patch versions as well as the newest, would exercise that path. For clusters with restricted egress, pinning the node image or moving to a Kubernetes version cached in the image were both quick workarounds while the fix was shipped.

   View case
2. IL-0078 Enabling EKS secrets encryption with KMS through Terraform removed worker node IAM permissions and took down all ZenML Pro workspaces

   ZenML2025

   Symptoms

   Between 12 AM and 8 AM CEST on 11 May 2025, every ZenML Pro workspace server was unreachable, so customers could not use workspace UIs or pipelines. The ZenML Pro dashboard was not affected.

   Service outage

   Trigger

   At 23:38 on 10 May, secrets encryption with AWS KMS was enabled through Terraform as part of the EKS module upgrade.

   Root cause

   Software defect

   Component

   TerraformNode pool

   Contributing factor

   Unrepresentative stagingDeferred upgrades

   Detection

   Engineers found the cluster degraded when they resumed the upgrade at 06:00.

   Remediation

   • deleted and recreated the EKS node group with the correct IAM permissions and configuration
   • restored service access across all affected ZenML Pro environments

   Action items

   • Upgrade all Terraform modules (EKS, VPC) to current supported versions
   • Complete the EKS cluster upgrades as planned
   • Review the strategy for periodic Kubernetes and infrastructure upgrades to prevent a recurrence

   Lessons

   A successful upgrade on a much smaller staging cluster understated the risk of the same change in production. Skipping Kubernetes and module versions let upgrade complexity build up until one upgrade carried many changes at once.

   View case
3. IL-0161 A faulty earlier API update made new LKE cluster provisioning in Toronto time out for three days, leaving clusters with no node pools or the wrong number of nodes

   Linode2024

   Symptoms

   At about 14:05 UTC on 22 April 2024, Linode's team noticed that new LKE clusters were failing to provision in the Toronto data center.

   Cluster or node provisioning stallIntermittent timeouts

   Trigger

   A previous update to the API did not work as intended and caused timeouts when new LKE clusters were provisioned.

   Root cause

   Software defect

   Component

   Node pool

   Contributing factor

   –

   Detection

   Linode's team noticed new LKE clusters failing to provision in the Toronto data center at about 14:05 UTC on 22 April 2024.

   Remediation

   • disabled the affected section of Toronto infrastructure from provisioning new LKE clusters
   • implemented a fix for the faulty API update
   • put the disabled infrastructure back into use after the fix

   Lessons

   The fault was in an earlier API change, not in the infrastructure that showed it, but most failures concentrated on one backend resource in one data center. Taking that resource out of provisioning limited the impact while the root cause was found, which took three days.

   View case
4. IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutes

   Grafana Labs2020

   Symptoms

   About a minute after the etcd leader changed, two replicas of the Cortex distributor started logging "context deadline exceeded" when they tried to read keys from etcd. Writes from some customers that landed on those two replicas failed, while the other distributor replicas kept working, so the service saw an error rate of about 20% on writes to the London region of Hosted Prometheus.

   High error rateIntermittent timeouts

   Trigger

   A scheduled Kubernetes version upgrade abruptly terminated the node that ran the etcd leader at 11:19:00 UTC.

   Root cause

   Software defect

   Component

   etcdNode pool

   Contributing factor

   Unreviewed defaultsNo safeguard on destructive change

   Detection

   An SLO-based alert paged the on-call engineer at 11:31, 11 minutes after the errors started. At a 20% error rate the monthly SLA would have taken about 18 hours to breach.

   Remediation

   • Restarted the two affected Cortex distributor replicas at 11:32, which stopped the errors

   Action items

   • Enable gRPC keepalive probes in the Cortex etcd client (DialKeepAliveTime 10s, DialKeepAliveTimeout, PermitWithoutStream)
   • Make the Kubernetes upgrade process terminate pods and machines gracefully, including the etcd pods managed by the etcd operator

   Lessons

   When a node dies abruptly, clients holding TCP connections to it are not told; without keepalive probes an etcd client can keep sending requests into a dead connection long after a new leader exists. A failover that works on the server side still needs clients that detect the dead peer. Node replacement during cluster upgrades should drain pods gracefully, including pods owned by operators that the upgrade scripts do not know about, and should give instances enough time to shut down.

   View case

By root cause1