Incident Library
Applied filters Root causeBreaking change in an upgradeComponentLinux kernelClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress traffic
    Altinity2025
    Symptoms

    On a Saturday, an OpsGenie alert fired for a ClickHouse cluster in Altinity.Cloud that had gone silent: no queries, no metrics, and no logs from customers trying to connect. The pods were running and the nodes looked healthy, but no traffic reached the database. Other clusters soon showed the same thing, and external access was blocked across several GKE clusters.

    Service outage
    Trigger

    GKE automatically upgraded worker nodes to the ubuntu_containerd image 1.32.3-gke.1785003, which ships kernel 6.8.0-1022-gke.

    Root cause
    Breaking change in an upgrade
    Component
    kube-proxyNode operating system imageLinux kernelCloud load balancer
    Contributing factor
    Unpatched known bugNo staged rollout
    Detection

    An OpsGenie alert fired when one ClickHouse cluster stopped receiving queries, metrics and customer connection logs.

    Remediation
    • added maintenance exceptions to pause all upgrades on the GKE clusters that were not yet affected
    • changed the load balancer health check policy to ignore failed kube-proxy health checks, so traffic flowed to the nodes again
    • began migrating the affected clusters from the ubuntu_containerd to the cos_containerd node image by recreating nodes
    Action items
    • Expand image pre-testing pipelines to catch regressions earlier
    • Enable notifications for upcoming GKE auto-upgrades
    • Make GKE maintenance windows required across the fleet to control when upgrades roll out
    • Track Google's issue and wait for a patched Ubuntu image
    Lessons

    A managed node upgrade can break kube-proxy in a way that leaves pods and nodes looking healthy while the load balancer sends nothing, because the load balancer health check depends on kube-proxy. kube-proxy's own logs showed the cause. Altinity recommends that GKE users on ubuntu_containerd restore traffic by switching to externalTrafficPolicy: Cluster, which disables the kube-proxy health check in the load balancer, move production workloads to cos_containerd, and pause auto-upgrades where possible until a patched image has been confirmed and tested.

    View case
  2. IL-0201 A framework upgrade caused kernel panics in GitHub's Kubernetes infrastructure, and GitHub.com web and API requests, mostly for issues and pull requests, failed intermittently for about 16 hours
    GitHub2025
    Symptoms

    From 18:05 UTC on March 17, 2025, GitHub.com web and API requests failed now and then for a small percentage of users, mostly requests about issues and pull requests. At 18:39 UTC the status page reported degraded performance for Issues.

    High error rateIntermittent timeouts
    Trigger

    A framework upgrade rolled out to GitHub's Kubernetes infrastructure caused kernel panics.

    Root cause
    Breaking change in an upgrade
    Component
    Linux kernel
    Contributing factor
    –
    Detection

    The status page opened the incident at 18:39 UTC, 34 minutes after the failures began, as an investigation of reports of degraded performance for Issues.

    Remediation
    • Downgraded the framework upgrade that caused the kernel panics
    • Disabled the problematic feature of the framework
    Action items
    • Investigate why the upgrade caused the kernel panics
    • Take steps to temporarily prevent the problem from coming back
    • Work out longer-term patch plans
    • Improve observability so that similar classes of problems can be reacted to quickly
    View case
Check your cluster for this

These 2 cases involve Linux kernel and Cloud load balancer. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade2 cases

Component
Linux kernel2Cloud load balancer1kube-proxy1Node operating system image1
Contributing factor
Unpatched known bug1No staged rollout1
Organisation
Altinity1GitHub1
  1. IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress traffic
    Altinity2025
  2. IL-0201 A framework upgrade caused kernel panics in GitHub's Kubernetes infrastructure, and GitHub.com web and API requests, mostly for issues and pull requests, failed intermittently for about 16 hours
    GitHub2025