Incident Library

Organisation

Wikimedia Foundation

Wikimedia Foundation · wikimedia-foundation

12 cases · 2019 to 2026

Root cause Configuration change5Capacity exhaustion2Dependency failure1Faulty deployment1Hardware failure1Software defect1Breaking change in an upgrade1
Component Database4Helm3BGP routing2In-memory cache2Database replication2Elasticsearch or OpenSearch2Backbone network (WAN)1Calico1CDN and edge network1cert-manager1CI system1Configuration management tool1ConfigMaps and Secrets1etcd1Model inference server1Istio1Jobs and CronJobs1Kubernetes API server1
Contributing factor Unverified assumption4No runbook4Change with a delayed effect3Harmful manual action3Ignored early signal2Dependence on a warm cache2Signal buried in noise1Untested backup1Monitoring blind spot1No alert on limit1No configuration tests1Pages not delivered1Ineffective rollback1Single point of failure1Gap in automated tests1
Symptom High error rate6High latency4Control plane lag2Network unreachable2Replication lag2Service outage2TLS certificate errors2Writes rejected2Control plane unresponsive1Container crash loop1Data loss1Database overload1Log flood1Monitoring outage1Node disk full1Probe failures1Misrouted requests1Load imbalance1

Cases

  1. IL-0236 After an Istio 1.15 to 1.24 upgrade, half of Wikimedia's ML model servers returned 5XX for about a dayWikimedia Foundation · 2026

    Root cause Breaking change in an upgrade

  2. IL-0299 About 60 indices of Wikimedia's CirrusSearch OpenSearch clusters went red and unwritable during a host decommissionWikimedia Foundation · 2025

    Root cause Configuration change

  3. IL-0298 Wikimedia edits failed for 38 minutes when sessionstore Cassandra nodes in both data centres crashed on full disksWikimedia Foundation · 2025

    Root cause Capacity exhaustion

  4. IL-0324 Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database clusterWikimedia Foundation · 2025

    Root cause Faulty deployment

  5. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by oneWikimedia Foundation · 2024

    Root cause Capacity exhaustion

  6. IL-0392 MariaDB replication of section x1 between eqiad and codfw broke on a missing heartbeat row during a cookbook runWikimedia Foundation · 2024

    Root cause Configuration change

  7. IL-0216 Wikimedia on-wiki search was slow or rejected after a datacenter switchover overloaded one Elasticsearch node in codfwWikimedia Foundation · 2023

    Root cause Software defect

  8. IL-0459 MediaWiki on Kubernetes terminated TLS with a wrong certificate after a deployment, and 1% of requests failedWikimedia Foundation · 2023

    Root cause Configuration change

  9. IL-0267 Wikimedia's eqsin cache site went unreachable when both of its transport links to the core data centers were downWikimedia Foundation · 2023

    Root cause Dependency failure

  10. IL-0266 Eqiad routers looped traffic between each other after a BGP MED change and could not reach other sites' public IPsWikimedia Foundation · 2021

    Root cause Configuration change

  11. IL-0323 A RAID battery backup unit failure crashed db1075, the MariaDB primary of section s3, and every s3 wiki went read-onlyWikimedia Foundation · 2019

    Root cause Hardware failure

  12. IL-0458 Toolforge Kubernetes API server failed to start after a restart, logging "etcd cluster is unavailable or misconfigured"Wikimedia Foundation · 2019

    Root cause Configuration change