Incident Library
Applied filters Root causeBreaking change in an upgradeComponentkube-proxyDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress traffic
    Altinity2025
    Symptoms

    On a Saturday, an OpsGenie alert fired for a ClickHouse cluster in Altinity.Cloud that had gone silent: no queries, no metrics, and no logs from customers trying to connect. The pods were running and the nodes looked healthy, but no traffic reached the database. Other clusters soon showed the same thing, and external access was blocked across several GKE clusters.

    Service outage
    Trigger

    GKE automatically upgraded worker nodes to the ubuntu_containerd image 1.32.3-gke.1785003, which ships kernel 6.8.0-1022-gke.

    Root cause
    Breaking change in an upgrade
    Component
    kube-proxyNode operating system imageLinux kernelCloud load balancer
    Contributing factor
    Unpatched known bugNo staged rollout
    Detection

    An OpsGenie alert fired when one ClickHouse cluster stopped receiving queries, metrics and customer connection logs.

    Remediation
    • added maintenance exceptions to pause all upgrades on the GKE clusters that were not yet affected
    • changed the load balancer health check policy to ignore failed kube-proxy health checks, so traffic flowed to the nodes again
    • began migrating the affected clusters from the ubuntu_containerd to the cos_containerd node image by recreating nodes
    Action items
    • Expand image pre-testing pipelines to catch regressions earlier
    • Enable notifications for upcoming GKE auto-upgrades
    • Make GKE maintenance windows required across the fleet to control when upgrades roll out
    • Track Google's issue and wait for a patched Ubuntu image
    Lessons

    A managed node upgrade can break kube-proxy in a way that leaves pods and nodes looking healthy while the load balancer sends nothing, because the load balancer health check depends on kube-proxy. kube-proxy's own logs showed the cause. Altinity recommends that GKE users on ubuntu_containerd restore traffic by switching to externalTrafficPolicy: Cluster, which disables the kube-proxy health check in the load balancer, move production workloads to cos_containerd, and pause auto-upgrades where possible until a patched image has been confirmed and tested.

    View case
  2. IL-0230 kube-proxy sync failures during a Kubernetes upgrade corrupted node iptables, and CircleCI jobs stopped starting
    CircleCI2023
    Symptoms

    On 14 March 2023 at about 18:00 UTC, communication errors rose on one CircleCI service, and pipelines for all customers began to start late until jobs did not run at all. Within an hour the errors had spread to many services and to every function of the application, including the CircleCI UI and API.

    Service outageIntermittent connection failuresDNS resolution failures
    Trigger

    At about 17:40 UTC on 14 March the infrastructure team resumed the upgrade of the main production Kubernetes cluster with a changed approach for some components, after a first attempt on 13 March had been rolled back. The ongoing upgrade left kubelet and kube-proxy at mismatched versions.

    Root cause
    Breaking change in an upgrade
    Component
    kube-proxykubeletCoreDNSClient-side DNS resolver and cache
    Contributing factor
    Partial rolloutIneffective rollbackMonitoring blind spotHarmful manual action
    Detection

    At 18:00 UTC engineers saw communication errors rise on one service, then job processing delays climbing and job queues backing up across several services. The kube-proxy "Sync failed" errors were found only in the analysis after the incident.

    Remediation
    • Restarted every node in the cluster so each node built a fresh iptables ruleset
    • Restarted a separate pool of dependencies so jobs using contexts or the machine executor ran again
    • Added capacity to work through the backlog of jobs
    • Restarted the RabbitMQ nodes to fix a few corrupted queues
    Action items
    • Improve the Kubernetes upgrade process to limit the risk of future upgrades
    • Add metrics and alerts for kube-proxy sync errors
    • Evaluate ways to make the RabbitMQ infrastructure more resilient
    Lessons

    No single trigger caused the incident. Because networking underlies everything in the cluster, Kubernetes upgrades carry inherent risk, and CircleCI's risk mitigation did not hold up when several events interacted.

    View case
Check your cluster for this

These 2 cases involve kube-proxy and Cloud load balancer. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade2 cases

Component
kube-proxy2Cloud load balancer1CoreDNS1Client-side DNS resolver and cache1kubelet1Linux kernel1Node operating system image1
Contributing factor
Unpatched known bug1Harmful manual action1Monitoring blind spot1No staged rollout1Partial rollout1Ineffective rollback1
Organisation
Altinity1CircleCI1
  1. IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress traffic
    Altinity2025
  2. IL-0230 kube-proxy sync failures during a Kubernetes upgrade corrupted node iptables, and CircleCI jobs stopped starting
    CircleCI2023