Incident Library
Applied filters ComponentClient-side DNS resolver and cacheFactorPartial rolloutClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0230 kube-proxy sync failures during a Kubernetes upgrade corrupted node iptables, and CircleCI jobs stopped starting
    CircleCI2023
    Symptoms

    On 14 March 2023 at about 18:00 UTC, communication errors rose on one CircleCI service, and pipelines for all customers began to start late until jobs did not run at all. Within an hour the errors had spread to many services and to every function of the application, including the CircleCI UI and API.

    Service outageIntermittent connection failuresDNS resolution failures
    Trigger

    At about 17:40 UTC on 14 March the infrastructure team resumed the upgrade of the main production Kubernetes cluster with a changed approach for some components, after a first attempt on 13 March had been rolled back. The ongoing upgrade left kubelet and kube-proxy at mismatched versions.

    Root cause
    Breaking change in an upgrade
    Component
    kube-proxykubeletCoreDNSClient-side DNS resolver and cache
    Contributing factor
    Partial rolloutIneffective rollbackMonitoring blind spotHarmful manual action
    Detection

    At 18:00 UTC engineers saw communication errors rise on one service, then job processing delays climbing and job queues backing up across several services. The kube-proxy "Sync failed" errors were found only in the analysis after the incident.

    Remediation
    • Restarted every node in the cluster so each node built a fresh iptables ruleset
    • Restarted a separate pool of dependencies so jobs using contexts or the machine executor ran again
    • Added capacity to work through the backlog of jobs
    • Restarted the RabbitMQ nodes to fix a few corrupted queues
    Action items
    • Improve the Kubernetes upgrade process to limit the risk of future upgrades
    • Add metrics and alerts for kube-proxy sync errors
    • Evaluate ways to make the RabbitMQ infrastructure more resilient
    Lessons

    No single trigger caused the incident. Because networking underlies everything in the cluster, Kubernetes upgrades carry inherent risk, and CircleCI's risk mitigation did not hold up when several events interacted.

    View case
Check your cluster for this

This case involves CoreDNS and Client-side DNS resolver and cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade1 case

Component
CoreDNSClient-side DNS resolver and cachekube-proxykubelet
Contributing factor
Harmful manual actionMonitoring blind spotPartial rolloutIneffective rollback
Organisation
CircleCI
  1. IL-0230 kube-proxy sync failures during a Kubernetes upgrade corrupted node iptables, and CircleCI jobs stopped starting
    CircleCI2023