Incident Library

IL-0230 CircleCI · 2023-03-14kubernetesnetworking

kube-proxy sync failures during a Kubernetes upgrade corrupted node iptables, and CircleCI jobs stopped starting

In the week before the incident the CircleCI infrastructure team was upgrading its main production Kubernetes cluster step by step. On 13 March, after a few components were updated, some services reported missing metrics; the team reverted the changes and the metrics came back. The team adjusted the plan to limit the blast radius and at about 17:40 UTC on 14 March tried a different way of upgrading those components. Between the two Kubernetes versions, the way kube-proxy handles and formats its iptables rulesets had changed. kube-proxy rewrites each node's iptables rules with iptables-save and iptables-restore in Proxier.syncProxyRules() whenever Service and Endpoints objects change. After the kube-proxy rollback on 13 March a small number of sync failures had already started. As the upgrade rolled out more pods and Endpoints changed, the failures grew, and on 14 March a spike of sync errors left iptables on many nodes badly corrupted, so traffic between services failed from about 18:00 UTC.

With all three suspected networking workloads looking healthy, the team restarted node-local-dns-cache and CoreDNS, which made DNS resolution fail completely. A single node restart showed that traffic through a freshly restarted node worked, so the team restarted every node in the cluster; each restart built a fresh iptables ruleset and the sync errors went away. By 23:00 UTC all nodes had been rolled and jobs recovered except those using contexts or the machine executor. After a separate pool of dependencies was restarted, all jobs ran correctly by 01:15 UTC on 15 March, and the incident is given as lasting until about 01:27 UTC.

The recovery caused two further incidents in the RabbitMQ queuing system. Logging for RabbitMQ was found missing, so the team rolled the rest of the infrastructure; at about 02:50 UTC RabbitMQ connection errors delayed jobs again, mitigated by 03:35 UTC and resolved by 05:14 UTC. The report ties a later restart of the RabbitMQ nodes, done to fix a few corrupted queues, to a further degradation from 09:50 UTC in which GitHub checks were delayed by up to 30 minutes; it was mitigated by 11:08 UTC and resolved at 13:15 UTC.

Severity
–
Scope
Pipelines of all customers started late and then jobs did not run; Docker, Machine, macOS and Windows jobs, the CircleCI UI and Runner were affected. Two follow-on RabbitMQ incidents delayed all pipelines from 02:50 to 05:14 UTC and GitHub checks by up to 30 minutes from 09:50 to 13:15 UTC on 15 March.
Users affected
all customers
Region
–
Time to detect
–
Time to resolve
7h 27m

Symptoms

On 14 March 2023 at about 18:00 UTC, communication errors rose on one CircleCI service, and pipelines for all customers began to start late until jobs did not run at all. Within an hour the errors had spread to many services and to every function of the application, including the CircleCI UI and API.

Job processing delays kept climbing and job queues backed up across several services. A few services were restarted to ease some of the symptoms. The common thread turned out to be failed service-to-service communication inside the main production Kubernetes cluster. Engineers searched logs and metrics and tried to reach services by their internal Kubernetes domain names from inside pods, and suspected kube-proxy, CoreDNS or node-local-dns-cache, but all three workloads looked healthy.

Between 20:00 and 21:00 UTC the team restarted node-local-dns-cache and CoreDNS, and DNS resolution then failed completely. Restarting one node made requests through that node work again, which pointed at kube-proxy. After every node had been restarted by 23:00 UTC, jobs ran again except those using contexts or the machine executor.

The analysis afterwards found that kube-proxy, which runs on every node, had a spike of sync errors at the start of the incident, logged as "Failed to execute iptables-restore" err="exit status 1 (iptables-restore: line 8022 failed\n)" followed by "Sync failed" retryingTime="30s". A small number of the same errors had already appeared on 13 March, after a first upgrade attempt had made some services report missing metrics and had been rolled back, but nobody noticed them then.

Symptom Service outageIntermittent connection failuresDNS resolution failures

Root cause

Root cause Breaking change in an upgrade

Component kube-proxykubeletCoreDNSClient-side DNS resolver and cache
Contributing factor Partial rolloutIneffective rollbackMonitoring blind spotHarmful manual action
Trigger
At about 17:40 UTC on 14 March the infrastructure team resumed the upgrade of the main production Kubernetes cluster with a changed approach for some components, after a first attempt on 13 March had been rolled back. The ongoing upgrade left kubelet and kube-proxy at mismatched versions.

Detection

At 18:00 UTC engineers saw communication errors rise on one service, then job processing delays climbing and job queues backing up across several services. The kube-proxy "Sync failed" errors were found only in the analysis after the incident.

Remediation

Action items

Lessons

No single trigger caused the incident. Because networking underlies everything in the cluster, Kubernetes upgrades carry inherent risk, and CircleCI's risk mitigation did not hold up when several events interacted.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, kube-proxy, kubelet, CoreDNS among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.