Incident Library
Applied filters Root causeBreaking change in an upgradeComponentNode operating system imageSymptomService outageClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress traffic
    Altinity2025
    Symptoms

    On a Saturday, an OpsGenie alert fired for a ClickHouse cluster in Altinity.Cloud that had gone silent: no queries, no metrics, and no logs from customers trying to connect. The pods were running and the nodes looked healthy, but no traffic reached the database. Other clusters soon showed the same thing, and external access was blocked across several GKE clusters.

    Service outage
    Trigger

    GKE automatically upgraded worker nodes to the ubuntu_containerd image 1.32.3-gke.1785003, which ships kernel 6.8.0-1022-gke.

    Root cause
    Breaking change in an upgrade
    Component
    kube-proxyNode operating system imageLinux kernelCloud load balancer
    Contributing factor
    Unpatched known bugNo staged rollout
    Detection

    An OpsGenie alert fired when one ClickHouse cluster stopped receiving queries, metrics and customer connection logs.

    Remediation
    • added maintenance exceptions to pause all upgrades on the GKE clusters that were not yet affected
    • changed the load balancer health check policy to ignore failed kube-proxy health checks, so traffic flowed to the nodes again
    • began migrating the affected clusters from the ubuntu_containerd to the cos_containerd node image by recreating nodes
    Action items
    • Expand image pre-testing pipelines to catch regressions earlier
    • Enable notifications for upcoming GKE auto-upgrades
    • Make GKE maintenance windows required across the fleet to control when upgrades roll out
    • Track Google's issue and wait for a patched Ubuntu image
    Lessons

    A managed node upgrade can break kube-proxy in a way that leaves pods and nodes looking healthy while the load balancer sends nothing, because the load balancer health check depends on kube-proxy. kube-proxy's own logs showed the cause. Altinity recommends that GKE users on ubuntu_containerd restore traffic by switching to externalTrafficPolicy: Cluster, which disables the kube-proxy health check in the load balancer, move production workloads to cos_containerd, and pause auto-upgrades where possible until a patched image has been confirmed and tested.

    View case
  2. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023
    Symptoms

    Between 06:00 and 07:00 UTC on 8 March 2023, Kubernetes nodes in several Datadog regions lost network connectivity at the same time, on AWS, Google Cloud and Azure. All services across multiple regions were affected, including the regions that host Datadog's CI/CD and automation tools. Datadog's regions are not connected to each other and changes are rolled out one region at a time, so a simultaneous failure on several cloud providers looked implausible at first.

    Node NotReadyService outageThroughput degradation
    Trigger

    On 7 March 2023 Ubuntu published a systemd patch for a CVE (249.11-0ubuntu3.7). Ubuntu's default unattended upgrades installed it on the nodes between 06:00 and 07:00 UTC on 8 March, and installing it restarted systemd-networkd.

    Root cause
    Breaking change in an upgrade
    Component
    systemd-networkdCiliumNode operating system image
    Contributing factor
    Unreviewed defaultsChange with a delayed effectNo staged rolloutAutomation amplification
    Remediation
    • restarted thousands of disconnected instances on Google Cloud and Azure through the cloud APIs, keeping their local disks
    • rebuilt the data lost with terminated AWS instances' local disks through replication, reprocessing and restores from backup
    Lessons

    A change in a base operating system component can stay dormant for months until something restarts that component. Here the trigger was a security patch, and the change reached every region at once because automatic OS updates ran outside the progressive, region-by-region process used for every other change. Cloud providers react differently to an instance that loses its network. Google Cloud and Azure left the instances running, so a restart recovered them with their disks; AWS replaced them quickly, which brought stateless services back fast but destroyed the local-disk data the stateful services needed.

    View case
Check your cluster for this

These 2 cases involve Node operating system image and Cilium. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade2 cases

Component
Node operating system image2Cilium1Cloud load balancer1kube-proxy1Linux kernel1systemd-networkd1
Contributing factor
No staged rollout2Automation amplification1Unreviewed defaults1Change with a delayed effect1Unpatched known bug1
Organisation
Altinity1Datadog1
  1. IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress traffic
    Altinity2025
  2. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023