Cases12 cases

1 to 5 of 12

1. IL-0236 After an Istio 1.15 to 1.24 upgrade, half of Wikimedia's ML model servers returned 5XX for about a day

   Wikimedia Foundation2026

   Symptoms

   From 17:30 UTC on 23 February 2026, right after the ml-serve-codfw Kubernetes cluster was upgraded, 5XX responses appeared in the metrics of a subset of Wikimedia's machine learning model servers and alerts started to fire. About half of the running model servers were affected and stayed unreachable for roughly 24 hours.

   High error rate

   Trigger

   The two production Machine Learning Kubernetes clusters were upgraded to version 1.31, which included upgrading Istio from 1.15 to 1.24; ml-serve-codfw was done at 17:21 UTC on 23 February and ml-serve-eqiad between 08:00 and 11:00 UTC on 24 February.

   Root cause

   Breaking change in an upgrade

   Component

   IstioModel inference server

   Contributing factor

   Ignored early signal

   Detection

   Users of the affected model servers first reported the issue in the talk-to-machine-learning Slack channel. Alerts for HTTP 5XX had fired in the Machine Learning IRC channel right after the codfw upgrade, but nobody on the team noticed them.

   Remediation

   • Deployed an Istio hot patch that restored the broken functionality and fixed most of the HTTP 500 errors
   • Deployed the permanent fix to all affected clusters after verifying it on ml-staging-codfw

   Action items

   • Deliver alerts to the relevant Slack channels and watch IRC or alerts.wikimedia.org
   • Add more SLOs and fix the existing ones, whose Istio metrics changed

   Lessons

   The problem could have been caught on the staging cluster first, or by ad-hoc testing within the team after each cluster upgrade, which had been requested when the upgrades were announced. Not everyone involved knew that httpbb tests existed, and nobody on the Machine Learning team saw the alerts that fired right after the codfw upgrade. Recovery was fairly quick once users reported the problem, helped by having the right person online to find the Istio hot patch.

   View case
2. IL-0299 About 60 indices of Wikimedia's CirrusSearch OpenSearch clusters went red and unwritable during a host decommission

   Wikimedia Foundation2025

   Symptoms

   From 21:08 on 28 May 2025, about 60 indices of Wikimedia's OpenSearch clusters for CirrusSearch were in red, unwritable state. At 21:27 CirrusStreamingUpdaterFlinkJobUnstable alerts began to fire because the search update pipeline could not write to those indices.

   Writes rejected

   Trigger

   A Puppet patch that removed cluster access for the hosts elastic/cirrussearch10[53-67] while some of them still held primary shards.

   Root cause

   Configuration change

   Component

   Elasticsearch or OpenSearch

   Contributing factor

   Unverified assumptionNo runbookUntested backup

   Detection

   CirrusStreamingUpdaterFlinkJobUnstable alerts fired from 21:27; the review records the detection as human, by the engineer who started troubleshooting the next morning, and many other alerts were suppressed for maintenance.

   Remediation

   • Deleted the red indices, which brought the cluster back to green
   • Restored the broken indices

   Action items

   • Re-enable the snapshot repositories in production CirrusSearch
   • Create an operations-focused OpenSearch dashboard
   • Verify or fix the Logstash pipeline for Search Platform-owned OpenSearch clusters

   Lessons

   Banning nodes before removing them is a best-effort action and did not work as expected. A large operation such as decommissioning many servers at once had no maintenance plan or runbook, and the snapshots, dashboards and logging that recovery needed were not available or not documented well enough.

   View case
3. IL-0298 Wikimedia edits failed for 38 minutes when sessionstore Cassandra nodes in both data centres crashed on full disks

   Wikimedia Foundation2025

   Symptoms

   At 02:58 UTC on 31 March 2025, the sessionstore service in both Wikimedia data centres began returning 500 errors and edits failed. MediaWikiEditFailures (session loss) fired as critical at 03:03, and SessionStoreErrorRateHigh paged at 03:04.

   High error rateService outageNode disk fullData loss

   Trigger

   A rise in session writes, made up of session overwrites at a high rate, filled the storage of the sessionstore Cassandra nodes. The review says the rollout of SUL3 correlates strongly with it.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No alert on limitUnverified assumption

   Detection

   MediaWikiEditFailures (session loss) fired as critical at 03:03 UTC, and SessionStoreErrorRateHigh, a high 5xx rate alert, paged the batphone at 03:04. Users also reported the problem in task T390512.

   Remediation

   • Wiped the sessionstore storage, removing all sessions, and restarted Cassandra

   Action items

   • Page when the disk utilisation trend becomes unreasonable
   • Increase sessionstore storage capacity
   • Add namespacing to sessionstore
   • Add observability of the sessionstore workload
   • Identify the cause of the high rate of storage growth in sessionstore Cassandra
   • Consider an alert, without paging, on the Cassandra error rate
   • Consider paging or alerting when Cassandra nodes enter a down state

   Lessons

   The storage growth before the node failures made their timing predictable, so an alert on high storage utilisation well in advance would have given a chance to act before users were affected, and an alert on Cassandra node failures would have told responders more than an HTTP error rate alert. A cluster seen as heavily over-provisioned can still grow quickly and unsustainably under an unusual workload.

   View case
4. IL-0324 Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database cluster

   Wikimedia Foundation2025

   Symptoms

   On 2025-03-12 from 10:58 UTC, alerts for high backend response times came in, and the MediaWiki deployments mw-api-int, mw-parsoid and mw-jobrunner, which serve MediaWiki for non-user-facing purposes, showed high latency and database errors. The External Storage cluster, whose hosts store the compressed text of wiki page revisions, had a sharp rise in load and connections. The same happened again on 2025-03-13 from 11:15 UTC.

   Database overloadHigh latencyHigh error rateLog floodMonitoring outage

   Trigger

   At 10:44 UTC on 12 March a Scap deployment of the PHP 8.1 rollout for mw-api-int, mw-parsoid and mw-jobrunner was run with the flag -Dbuild_mw_container_image:False; the rollout was deployed with Scap again at 11:15 UTC on 13 March.

   Root cause

   Faulty deployment

   Component

   DatabaseIn-memory cacheConfigMaps and SecretsHelmCI system

   Contributing factor

   Unverified assumptionDependence on a warm cacheSignal buried in noise

   Detection

   Alerts for high backend response times started at 10:58 UTC on 12 March, 14 minutes after the Scap deployment; 2 on-call engineers were paged. The MediaWikiMemcachedHighErrorRate alert that pointed at the cause fired only briefly because its log pipeline was overloaded.

   Remediation

   • Deployed a patch to cirrus-streaming-updater that reduced SUP parallelism
   • Reduced the concurrency of the categoryMembershipChange job in changeprop-jobqueue
   • On 12 March, reverted 1126607 and 1126650 and re-ran Scap
   • On 13 March, rolling-restarted changeprop-jobqueue
   • Rolling-restarted mw-api-int
   • Disabled the categorymembership job in changeprop-jobqueue
   • Reverted 1127476 and 1127478
   • Redeployed mw-api-int to pick up the revert
   • Increased the number of mw-api-int replicas
   • Redeployed mw-jobrunner to pick up the revert
   • Redeployed mw-parsoid to pick up the revert

   Action items

   • Reduce the number of messages sent through channel:Memcached during failures
   • Add helm rollback functionality to Scap
   • Decide whether Scap should update helmfile-defaults when -Dbuild_mw_container_image:False is set
   • Consider removing envvars.inc from MediaWiki images
   • Consider "Why" dashboards that give an overview of a group of related components together with relevant links to follow
   • Coordinate incident responders better
   • Be able to work with MediaWiki developers during incidents

   Lessons

   The write-up puts the weight not on the technical root cause but on why the team could not see, even after the 12 March incident, that three MediaWiki deployments were running without Memcached. Alert dashboards showed what was happening, such as many busy PHP workers, but not why; dashboards linking components that affect each other, such as SQLBlobStore keys and External Storage, would have helped. The alert that pointed in the right direction did not stay firing and would probably have been lost among 78 others. The incident coordinator could not tell who was working on what or who had joined or left, the channels were noisy, the external impact was judged more critical than it was, and trace.wikimedia.org was not used although it might have helped.

   View case
5. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one

   Wikimedia Foundation2024

   Symptoms

   At 05:01 UTC on 9 October 2024 a page fired for GatewayBackendErrorsHigh, elevated 5xx errors from page-analytics_cluster in codfw, followed by a growing number of KubernetesCalicoDown alerts for WikiKube nodes not running the calico-node pod and by MediaWikiMemcachedHighErrorRate. For about an hour uncached MediaWiki requests, such as those of logged-in users, edits and API calls, saw high latency or errors.

   High error rateHigh latencyContainer crash loopProbe failuresControl plane lag

   Trigger

   An operator used the new mwscript-k8s utility to run a batch of video transcode maintenance, the TimedMediaHandler requeueTranscodes.php script. mwscript-k8s deploys a whole helm release for every script invocation, so the batch created close to 2k Calico NetworkPolicies along with many other resources over several hours.

   Root cause

   Capacity exhaustion

   Component

   CalicoHelmJobs and CronJobs

   Contributing factor

   Change with a delayed effectNo runbook

   Detection

   Automated monitoring detected the issue and paged the on-call SRE: GatewayBackendErrorsHigh at 05:01:36 UTC, 16 KubernetesCalicoDown alerts at 05:03:32 UTC and MediaWikiMemcachedHighErrorRate at 05:04:15 UTC.

   Remediation

   • Raised the resource limits of the Calico and Typha pods and applied the change with helmfile, which redeployed all Calico pods

   Action items

   • Stop mwscript-k8s from creating so many resources

   Lessons

   The incident came close to a full outage. Automated monitoring caught it and many responders found the cause, but end users were affected and responders were unsure whether they could switch over to eqiad. The problem was noticed just before the hand-off from Americas to EMEA SRE shifts, a period with a long-known lack of coverage, and it helped that the transcodes were not queued on a Friday evening in the US.

   View case

By root cause7