Incident Library

IL-0232 Wikimedia Foundation · 2024-10-09kubernetesnetworking

Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one

Wikimedia's mwscript-k8s utility runs MediaWiki maintenance scripts on the WikiKube Kubernetes cluster and, by design, deploys an entire helm release for each script invocation. An operator used it for a batch of video transcode requeues. The batch ran without trouble for hours while the number of resources it created grew steadily: close to 2k Calico NetworkPolicies, 1791 jobs in the mw-script namespace and a doubling of ConfigMaps in etcd. Every Calico component in WikiKube had to process the growing number of NetworkPolicies, and some of them reached their resource limits, were throttled and then failed. calico-node pods crash-looped or stopped being ready, the Calico controller's API calls timed out, and hardware nodes in codfw dropped out of service one after another. MediaWiki nodes in the primary datacenter were overloaded and uncached requests saw high latency and errors from 05:01 UTC.

Responders saw the Calico and Typha pods at their resource limits and prepared a change to raise them. While it was being applied with helmfile at about 06:05 UTC, the team also discussed moving traffic for everything except MediaWiki read-write to eqiad but was unsure whether a switchover was possible. The helmfile apply replaced the Calico pods with new healthy ones, 212 calico-node pods were ready by 06:07 UTC, and Calico had recovered by 06:10 UTC. One responder noted that the recovery may have come from redeploying all Calico pods at once rather than from the higher limits. Only afterwards did the team link the growth of objects to the mwscript-k8s jobs and find the requeueTranscodes.php invocation behind them. A responder linked the similar earlier incident "2024-04-03 calico/typha down", and the incident scorecard records that this incident was not different enough from earlier ones to be ruled out as a repeat.

Severity
–
Scope
For about one hour after 05:00 UTC some uncached MediaWiki requests, such as those from logged-in users, edits and API calls, saw high latency or errors from overloaded MediaWiki nodes in the primary datacenter
Users affected
Logged-in users, editors and API clients making uncached requests
Region
codfw
Time to detect
–
Time to resolve
1h 8m

Symptoms

At 05:01 UTC on 9 October 2024 a page fired for GatewayBackendErrorsHigh, elevated 5xx errors from page-analytics_cluster in codfw, followed by a growing number of KubernetesCalicoDown alerts for WikiKube nodes not running the calico-node pod and by MediaWikiMemcachedHighErrorRate. For about an hour uncached MediaWiki requests, such as those of logged-in users, edits and API calls, saw high latency or errors.

kubectl showed calico-kube-controllers in CrashLoopBackOff with 21 restarts. Its log ended in "Error getting cluster information config ClusterInformation="default" error=Get "https://10.192.72.1:443/apis/crd.projectcalico.org /v1/clusterinformations/default": context deadline exceeded" and then "[FATAL][1] main.go 124: Failed to initialize Calico datastore", so the Kubernetes API calls of the Calico controller were timing out. Typha looked available. calico-node pods failed their liveness probes and went into CrashLoopBackOff or stayed Running but not ready; their logs showed "bird: I/O loop cycle took 7808 ms for 6 events" and later "felix/health.go 211: Reporter is not ready. name="int_dataplane"". At 06:04 UTC only 25 nodes still had fully working calico-node pods.

Responders first asked whether etcd was the problem, found a rise in host CPU without any pegged machine and put it down to pod restarts, and saw nothing unusual in the Calico pod logs. A dashboard then showed the Calico and Typha pods hitting their resource limits. GET and LIST calls to the Kubernetes API had been rising since between 03:00 and 04:00 UTC, mostly for node and endpoint objects, and nodes running Typha saw a sharp rise in received traffic, about 7MB/s, from around 04:00 UTC. The effect on MediaWiki was hard to see on the application server dashboard: successful 200 responses fell by 25%, from 4k to 3k, for about five minutes, and latency rose briefly.

After recovery, ConfigMaps in the codfw etcd were found to have roughly doubled from 6k to 12k between 02:00 and 06:00 UTC, matching the count of jobs.batch objects and their network policies. `kubectl -n mw-script get jobs |wc -l` returned 1791, all younger than eight hours.

Symptom High error rateHigh latencyContainer crash loopProbe failuresControl plane lag

Root cause

Root cause Capacity exhaustion

Component CalicoHelmJobs and CronJobs
Contributing factor Change with a delayed effectNo runbook
Trigger
An operator used the new mwscript-k8s utility to run a batch of video transcode maintenance, the TimedMediaHandler requeueTranscodes.php script. mwscript-k8s deploys a whole helm release for every script invocation, so the batch created close to 2k Calico NetworkPolicies along with many other resources over several hours.

Detection

Automated monitoring detected the issue and paged the on-call SRE: GatewayBackendErrorsHigh at 05:01:36 UTC, 16 KubernetesCalicoDown alerts at 05:03:32 UTC and MediaWikiMemcachedHighErrorRate at 05:04:15 UTC.

Remediation

Action items

Lessons

The incident came close to a full outage. Automated monitoring caught it and many responders found the cause, but end users were affected and responders were unsure whether they could switch over to eqiad. The problem was noticed just before the hand-off from Americas to EMEA SRE shifts, a period with a long-known lack of coverage, and it helped that the transcodes were not queued on a Friday evening in the US.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Calico, Helm, Jobs and CronJobs among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.