Cases2 cases

1 to 2 of 2

1. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one

   Wikimedia Foundation2024

   Symptoms

   At 05:01 UTC on 9 October 2024 a page fired for GatewayBackendErrorsHigh, elevated 5xx errors from page-analytics_cluster in codfw, followed by a growing number of KubernetesCalicoDown alerts for WikiKube nodes not running the calico-node pod and by MediaWikiMemcachedHighErrorRate. For about an hour uncached MediaWiki requests, such as those of logged-in users, edits and API calls, saw high latency or errors.

   High error rateHigh latencyContainer crash loopProbe failuresControl plane lag

   Trigger

   An operator used the new mwscript-k8s utility to run a batch of video transcode maintenance, the TimedMediaHandler requeueTranscodes.php script. mwscript-k8s deploys a whole helm release for every script invocation, so the batch created close to 2k Calico NetworkPolicies along with many other resources over several hours.

   Root cause

   Capacity exhaustion

   Component

   CalicoHelmJobs and CronJobs

   Contributing factor

   Change with a delayed effectNo runbook

   Detection

   Automated monitoring detected the issue and paged the on-call SRE: GatewayBackendErrorsHigh at 05:01:36 UTC, 16 KubernetesCalicoDown alerts at 05:03:32 UTC and MediaWikiMemcachedHighErrorRate at 05:04:15 UTC.

   Remediation

   • Raised the resource limits of the Calico and Typha pods and applied the change with helmfile, which redeployed all Calico pods

   Action items

   • Stop mwscript-k8s from creating so many resources

   Lessons

   The incident came close to a full outage. Automated monitoring caught it and many responders found the cause, but end users were affected and responders were unsure whether they could switch over to eqiad. The problem was noticed just before the hand-off from Americas to EMEA SRE shifts, a period with a long-known lack of coverage, and it helped that the transcodes were not queued on a Friday evening in the US.

   View case
2. IL-0437 Old pods from an unfinished Argo Rollouts canary could not start with a Helm-updated ConfigMap and died in a cascade

   Gunosy2023

   Symptoms

   During a traffic surge from a breaking news alert, pods of a Gunosy service on Kubernetes could not start because the containers could not read their settings from the ConfigMap. The pods that kept running took the extra load and were restarted, failed to start again, and pod after pod died.

   Container crash loop

   Trigger

   A breaking news alert increased traffic and pods were scaled out automatically while a release that removed an old database's settings from the ConfigMap had been rolled out to only one canary pod.

   Root cause

   Faulty deployment

   Component

   ConfigMaps and SecretsHelmProgressive delivery controller

   Contributing factor

   Partial rolloutAutomation amplification

   Action items

   • Configure Argo Rollouts to roll out automatically, with setWeight 1, a one-hour pause and then setWeight 100
   • Add the "helm.sh/resource-policy": keep annotation so Helm keeps past versions of the ConfigMap
   • Build and use a Helm plugin, helm-clear, that keeps only the last few ConfigMap versions by the revision number in their names, set to keep 5 in CD

   Lessons

   Gunosy holds that a review that only resolves to be careful next time would soon be followed by the same failure, because human attention cannot be relied on, so each incident gets a permanent fix built into the system. Keeping past ConfigMap versions also made it possible to roll back quickly without going through CI/CD, which a ConfigMap change had previously prevented.

   View case

By root cause2