Cases6 cases

3 to 6 of 6

1. IL-0324 Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database cluster

   Wikimedia Foundation2025

   Symptoms

   On 2025-03-12 from 10:58 UTC, alerts for high backend response times came in, and the MediaWiki deployments mw-api-int, mw-parsoid and mw-jobrunner, which serve MediaWiki for non-user-facing purposes, showed high latency and database errors. The External Storage cluster, whose hosts store the compressed text of wiki page revisions, had a sharp rise in load and connections. The same happened again on 2025-03-13 from 11:15 UTC.

   Database overloadHigh latencyHigh error rateLog floodMonitoring outage

   Trigger

   At 10:44 UTC on 12 March a Scap deployment of the PHP 8.1 rollout for mw-api-int, mw-parsoid and mw-jobrunner was run with the flag -Dbuild_mw_container_image:False; the rollout was deployed with Scap again at 11:15 UTC on 13 March.

   Root cause

   Faulty deployment

   Component

   DatabaseIn-memory cacheConfigMaps and SecretsHelmCI system

   Contributing factor

   Unverified assumptionDependence on a warm cacheSignal buried in noise

   Detection

   Alerts for high backend response times started at 10:58 UTC on 12 March, 14 minutes after the Scap deployment; 2 on-call engineers were paged. The MediaWikiMemcachedHighErrorRate alert that pointed at the cause fired only briefly because its log pipeline was overloaded.

   Remediation

   • Deployed a patch to cirrus-streaming-updater that reduced SUP parallelism
   • Reduced the concurrency of the categoryMembershipChange job in changeprop-jobqueue
   • On 12 March, reverted 1126607 and 1126650 and re-ran Scap
   • On 13 March, rolling-restarted changeprop-jobqueue
   • Rolling-restarted mw-api-int
   • Disabled the categorymembership job in changeprop-jobqueue
   • Reverted 1127476 and 1127478
   • Redeployed mw-api-int to pick up the revert
   • Increased the number of mw-api-int replicas
   • Redeployed mw-jobrunner to pick up the revert
   • Redeployed mw-parsoid to pick up the revert

   Action items

   • Reduce the number of messages sent through channel:Memcached during failures
   • Add helm rollback functionality to Scap
   • Decide whether Scap should update helmfile-defaults when -Dbuild_mw_container_image:False is set
   • Consider removing envvars.inc from MediaWiki images
   • Consider "Why" dashboards that give an overview of a group of related components together with relevant links to follow
   • Coordinate incident responders better
   • Be able to work with MediaWiki developers during incidents

   Lessons

   The write-up puts the weight not on the technical root cause but on why the team could not see, even after the 12 March incident, that three MediaWiki deployments were running without Memcached. Alert dashboards showed what was happening, such as many busy PHP workers, but not why; dashboards linking components that affect each other, such as SQLBlobStore keys and External Storage, would have helped. The alert that pointed in the right direction did not stay firing and would probably have been lost among 78 others. The incident coordinator could not tell who was working on what or who had joined or left, the channels were noisy, the external impact was judged more critical than it was, and trace.wikimedia.org was not used although it might have helped.

   View case
2. IL-0437 Old pods from an unfinished Argo Rollouts canary could not start with a Helm-updated ConfigMap and died in a cascade

   Gunosy2023

   Symptoms

   During a traffic surge from a breaking news alert, pods of a Gunosy service on Kubernetes could not start because the containers could not read their settings from the ConfigMap. The pods that kept running took the extra load and were restarted, failed to start again, and pod after pod died.

   Container crash loop

   Trigger

   A breaking news alert increased traffic and pods were scaled out automatically while a release that removed an old database's settings from the ConfigMap had been rolled out to only one canary pod.

   Root cause

   Faulty deployment

   Component

   ConfigMaps and SecretsHelmProgressive delivery controller

   Contributing factor

   Partial rolloutAutomation amplification

   Action items

   • Configure Argo Rollouts to roll out automatically, with setWeight 1, a one-hour pause and then setWeight 100
   • Add the "helm.sh/resource-policy": keep annotation so Helm keeps past versions of the ConfigMap
   • Build and use a Helm plugin, helm-clear, that keeps only the last few ConfigMap versions by the revision number in their names, set to keep 5 in CD

   Lessons

   Gunosy holds that a review that only resolves to be careful next time would soon be followed by the same failure, because human attention cannot be relied on, so each incident gets a permanent fix built into the system. Keeping past ConfigMap versions also made it possible to roll back quickly without going through CI/CD, which a ConfigMap change had previously prevented.

   View case
3. IL-0185 A disk failure made the first control plane node of a Kubespray cluster read-only, and recovery needed the Kubespray inventory reordered and a new control plane node joined

   Hans H. Kim2023

   Symptoms

   The kernel on a master node of the Kubernetes cluster logged disk I/O errors: "[HAWKEYE] kernel:: [22329228.540857] blk_update_request: I/O error, dev sda, sector 795030104 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0". The sda disk switched to read-only, and the machine needed a new disk and an operating system reinstall.

   Node NotReady

   Trigger

   A hardware failure of the disk on the first control plane node.

   Root cause

   Hardware failure

   Component

   KubesprayetcdNode root diskConfigMaps and Secrets

   Contributing factor

   Single point of failure

   Detection

   Disk I/O errors from blk_update_request on dev sda appeared in the node's kernel log.

   Remediation

   • reordered the kube_control_plane group in the Kubespray inventory so that node-2 became the first control plane node
   • removed the failed machine from the cluster
   • edited the cluster-info ConfigMap in kube-public to point at a live control plane node and updated certificate-authority-data
   • installed and configured kubelet and the other components on a new node and joined it as a control plane node with kubeadm, skipping the etcd join phase
   • added the etcd member separately

   Lessons

   In a Kubespray cluster the first node of the kube_control_plane inventory group is special, so its failure needs its own procedure: move another node into the first position, fix the cluster-info ConfigMap, join a new control plane node and add the etcd member back. etcd should be brought back to full membership before another control plane node fails.

   View case
4. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods

   Pusher2019

   Symptoms

   During a cluster upgrade, all traffic to Pusher's products stopped. The speaker recalls that requests were failing with 500 errors.

   Service outageHigh error rate

   Trigger

   A cluster upgrade replaced nodes, so the bridge pods restarted and loaded a ConfigMap that had been changed three or four weeks earlier.

   Root cause

   Configuration change

   Component

   ConfigMaps and Secrets

   Contributing factor

   Change with a delayed effectUnverified assumption

   Action items

   • Build and run Wave so that changes to ConfigMaps and Secrets roll out new pods immediately

   Lessons

   A ConfigMap change does not restart the pods that use it, so for an application that cannot reload its configuration the change takes effect only when pods are next replaced, possibly weeks later during unrelated work. Tie configuration to the pod template so a bad change fails at the moment it is made.

   View case

By root cause4