Incident Library
Applied filters SymptomHigh error rateOrganisationWikimedia FoundationClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

5 to 6 of 6

  1. IL-0216 Wikimedia on-wiki search was slow or rejected after a datacenter switchover overloaded one Elasticsearch node in codfw
    Wikimedia Foundation2023
    Symptoms

    Right after the primary datacenter was switched from EQIAD to CODFW at 14:00 UTC on 20 September 2023, on-wiki search on all Wikimedia wikis became delayed or unavailable for a subset of users. At 14:08 the alert "CirrusSearch codfw 95th percentile latency - more_like" fired and reached IRC and the Search Platform email list as "PROBLEM alert - graphite1005/CirrusSearch codfw 95th percentile latency - more_like is CRITICAL".

    High latencyHigh error rateLoad imbalanceMisrouted requests
    Trigger

    At 14:00 UTC the primary datacenter was switched from EQIAD to CODFW. more_like queries that were expected to go to eqiad, the non-primary datacenter, went to codfw instead, apparently because RESTBase sent them as POST rather than GET.

    Root cause
    Software defect
    Component
    Elasticsearch or OpenSearchIn-memory cache
    Contributing factor
    Unverified assumptionDependence on a warm cacheNo runbook
    Detection

    Automated monitoring caught the problem, but not quickly enough to prevent user-facing impact: the Grafana alert "CirrusSearch codfw 95th percentile latency - more_like" went to IRC and the Search Platform email list at 14:08. No one was paged; SREs were already watching closely because of the datacenter switchover. At 14:14 Hashar pinged #wikimedia-search and engineers began troubleshooting.

    Remediation
    • Banned the overloaded node elastic2044 from the Elasticsearch cluster
    Action items
    • Work out which Elasticsearch steps are needed before a datacenter switchover
    • Update the Elasticsearch section of the datacenter switchover documentation
    • Review the suggested switchover improvements in T285347 and decide whether to implement them
    • Make the RESTBase /related/{article} endpoint send GET instead of POST so the CirrusSearch query cache stays warm in both datacenters
    • Review the shard balance of busy wikis, especially enwiki_content
    Lessons

    Monitoring caught the problem, but not quickly enough to prevent user-facing impact. The heightened attention during the switchover meant the impact was recognised immediately, and banning a single node was quick and easy. The mitigation was not guided by an existing runbook, and the shards on the overloaded host were not checked before it was banned.

    View case
  2. IL-0459 MediaWiki on Kubernetes terminated TLS with a wrong certificate after a deployment, and 1% of requests failed
    Wikimedia Foundation2023
    Symptoms

    On 9 August 2023, right after a MediaWiki configuration deployment at 11:41 UTC, the MediaWiki deployments on Kubernetes terminated TLS with a wrong certificate, and about 1% of Wikimedia web and API traffic, around 120 requests per second, received errors for about 30 minutes. The issue was detected by a page for failed service probes on mw-api-ext and mw-api-int.

    High error rateTLS certificate errorsControl plane lag
    Trigger

    At 09:37 a change to the global Kubernetes defaults was merged that switched supported charts to cert-manager certificates. The next MediaWiki on Kubernetes deployment, a configuration change at 11:41, picked it up.

    Root cause
    Configuration change
    Component
    cert-managerHelm
    Contributing factor
    Change with a delayed effectNo configuration tests
    Detection

    A page from the ProbeDown alerts for mw-api-ext:4447 and mw-api-int:4446 at 11:47; a KubernetesAPILatency alert had fired at 11:46.

    Remediation
    • Identified the issue and prepared a patch, https://gerrit.wikimedia.org/r/947331
    • Redeployed the MediaWiki deployments on Kubernetes
    Action items
    • Review whether deployment procedures and testing should be strengthened, for example against surprising changes on the next deployment and with canary deployments for Kubernetes
    • Look at metrics that become unavailable or unhealthy during deployments, or at reducing the impact of deployments
    Lessons

    The cause was found quickly because the person who made the breaking change was responding. The change should have been caught by CI, but the repository it was made in (hieradata) does not run the CI of the deployment charts. Only 1% of traffic going to Kubernetes kept the impact small, and the low error volume made it slower to link the errors to Kubernetes.

    View case
Check your cluster for this

These 6 cases involve Helm and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause5

Breaking change in an upgrade1 case

Component
Model inference serverIstio
Contributing factor
Ignored early signal
Organisation
Wikimedia Foundation
  1. IL-0236 After an Istio 1.15 to 1.24 upgrade, half of Wikimedia's ML model servers returned 5XX for about a day
    Wikimedia Foundation2026

Capacity exhaustion2 cases

Component
Calico1Database1Helm1Jobs and CronJobs1
Contributing factor
Unverified assumption1Change with a delayed effect1No alert on limit1No runbook1
Organisation
Wikimedia Foundation2
  1. IL-0298 Wikimedia edits failed for 38 minutes when sessionstore Cassandra nodes in both data centres crashed on full disks
    Wikimedia Foundation2025
  2. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one
    Wikimedia Foundation2024

Faulty deployment1 case

Component
In-memory cacheCI systemConfigMaps and SecretsDatabaseHelm
Contributing factor
Signal buried in noiseUnverified assumptionDependence on a warm cache
Organisation
Wikimedia Foundation
  1. IL-0324 Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database cluster
    Wikimedia Foundation2025

Software defect1 case

Component
In-memory cacheElasticsearch or OpenSearch
Contributing factor
Unverified assumptionNo runbookDependence on a warm cache
Organisation
Wikimedia Foundation
  1. IL-0216 Wikimedia on-wiki search was slow or rejected after a datacenter switchover overloaded one Elasticsearch node in codfw
    Wikimedia Foundation2023

Configuration change1 case

Component
cert-managerHelm
Contributing factor
Change with a delayed effectNo configuration tests
Organisation
Wikimedia Foundation
  1. IL-0459 MediaWiki on Kubernetes terminated TLS with a wrong certificate after a deployment, and 1% of requests failed
    Wikimedia Foundation2023