IL-0079 Altinity · 2025-05-17kubernetesgcpnetworking
Early on 17 May, several GKE clusters running Altinity.Cloud ClickHouse services lost all ingress traffic. GKE had auto-upgraded their worker nodes to the ubuntu_containerd image 1.32.3-gke.1785003 with kernel 6.8.0-1022-gke. That kernel lacked support for extensions kube-proxy uses, because of a regression in how the nf_tables module was packaged or loaded, so kube-proxy could not apply its IPv6 iptables rules. Its health check failed, the load balancer health checks that depend on it failed, and the load balancers stopped sending traffic to the nodes even though pods and nodes were otherwise healthy.
The regression was a known, ongoing problem in the Ubuntu kernel series, tracked in Ubuntu bug report #2101914, a Linux kernel mailing list thread and a community write-up. Altinity names four reasons it was hard to avoid: GKE auto-upgrades nodes with no way to opt out, IPv6 iptables rules are processed even when IPv6 is not used, GKE's image validation did not catch the problem, and the newer image builds had the same regression, so there was nothing working to roll forward to.
Altinity first added maintenance exceptions to pause upgrades on the clusters that were not yet affected, which kept the problem from spreading to all of its GKE clusters. Service on the affected clusters came back once the load balancer health check policy was changed to ignore failed kube-proxy health checks, without changing workloads or infrastructure. For a lasting fix Altinity started moving the affected clusters from ubuntu_containerd to the cos_containerd image family, whose kernel does not have the broken nf_tables behaviour; this meant recreating nodes and a small amount of downtime. At the time of writing GKE had not released a fixed Ubuntu image.
On a Saturday, an OpsGenie alert fired for a ClickHouse cluster in Altinity.Cloud that had gone silent: no queries, no metrics, and no logs from customers trying to connect. The pods were running and the nodes looked healthy, but no traffic reached the database. Other clusters soon showed the same thing, and external access was blocked across several GKE clusters.
The failures traced back to worker nodes that GKE had just auto-upgraded to the ubuntu_containerd image 1.32.3-gke.1785003 with kernel 6.8.0-1022-gke. Shortly after the upgrade, traffic to and from these nodes dropped to zero and the load balancer health checks against them failed.
kube-proxy logs showed it failing again and again to apply iptables rules, in particular for IPv6, with errors like "Warning: Extension MARK revision 0 not supported, missing kernel module?" and "ip6tables-restore v1.8.9 (nf_tables): unknown option "--xor-mark"". The workloads did not use IPv6, but GKE nodes process IPv6 iptables rules by default.
Rolling forward to the next available image, 1.32.3-gke.1927002, did not help: it had the same kernel problem.
Symptom Service outage
An OpsGenie alert fired when one ClickHouse cluster stopped receiving queries, metrics and customer connection logs.
A managed node upgrade can break kube-proxy in a way that leaves pods and nodes looking healthy while the load balancer sends nothing, because the load balancer health check depends on kube-proxy. kube-proxy's own logs showed the cause.
Altinity recommends that GKE users on ubuntu_containerd restore traffic by switching to externalTrafficPolicy: Cluster, which disables the kube-proxy health check in the load balancer, move production workloads to cos_containerd, and pause auto-upgrades where possible until a patched image has been confirmed and tested.
IL-0120 A daily scheduled OS update restarted systemd-networkd on cloud-hosted GPU nodes, which conflicted with OpenAI's networking agent and removed all routes, cutting capacity for ChatGPT and the APIOpenAI · 2025same root cause · shared Node operating system image
IL-0201 A framework upgrade caused kernel panics in GitHub's Kubernetes infrastructure, and GitHub.com web and API requests, mostly for issues and pull requests, failed intermittently for about 16 hoursGitHub · 2025same root cause · shared Linux kernel
IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replacedHoneycomb · 2024same root cause · shared Cloud load balancer
IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providersDatadog · 2023same root cause · shared Node operating system image
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Node operating system image