Cases2 cases

1 to 2 of 2

1. IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress traffic

   Altinity2025

   Symptoms

   On a Saturday, an OpsGenie alert fired for a ClickHouse cluster in Altinity.Cloud that had gone silent: no queries, no metrics, and no logs from customers trying to connect. The pods were running and the nodes looked healthy, but no traffic reached the database. Other clusters soon showed the same thing, and external access was blocked across several GKE clusters.

   Service outage

   Trigger

   GKE automatically upgraded worker nodes to the ubuntu_containerd image 1.32.3-gke.1785003, which ships kernel 6.8.0-1022-gke.

   Root cause

   Breaking change in an upgrade

   Component

   kube-proxyNode operating system imageLinux kernelCloud load balancer

   Contributing factor

   Unpatched known bugNo staged rollout

   Detection

   An OpsGenie alert fired when one ClickHouse cluster stopped receiving queries, metrics and customer connection logs.

   Remediation

   • added maintenance exceptions to pause all upgrades on the GKE clusters that were not yet affected
   • changed the load balancer health check policy to ignore failed kube-proxy health checks, so traffic flowed to the nodes again
   • began migrating the affected clusters from the ubuntu_containerd to the cos_containerd node image by recreating nodes

   Action items

   • Expand image pre-testing pipelines to catch regressions earlier
   • Enable notifications for upcoming GKE auto-upgrades
   • Make GKE maintenance windows required across the fleet to control when upgrades roll out
   • Track Google's issue and wait for a patched Ubuntu image

   Lessons

   A managed node upgrade can break kube-proxy in a way that leaves pods and nodes looking healthy while the load balancer sends nothing, because the load balancer health check depends on kube-proxy. kube-proxy's own logs showed the cause. Altinity recommends that GKE users on ubuntu_containerd restore traffic by switching to externalTrafficPolicy: Cluster, which disables the kube-proxy health check in the load balancer, move production workloads to cos_containerd, and pause auto-upgrades where possible until a patched image has been confirmed and tested.

   View case
2. IL-0030 Most nodes of a cluster went NotReady together for 15 minutes while the API server sat behind an AWS ELB

   Zalando2019

   Symptoms

   Fewer than 20% of the nodes in one Zalando cluster were available. Most nodes reported NotReady at the same moment, and only a handful stayed Ready.

   Node NotReady

   Root cause

   Software defect

   Component

   Kubernetes API serverCloud load balancer

   Contributing factor

   Unpatched known bug

   Remediation

   • none; the nodes recovered on their own after about 15 minutes

   View case

By root cause2