Graph
Cases5 cases
1 to 5 of 5
- Symptoms
From 21:08 on 28 May 2025, about 60 indices of Wikimedia's OpenSearch clusters for CirrusSearch were in red, unwritable state. At 21:27 CirrusStreamingUpdaterFlinkJobUnstable alerts began to fire because the search update pipeline could not write to those indices.
- Trigger
A Puppet patch that removed cluster access for the hosts elastic/cirrussearch10[53-67] while some of them still held primary shards.
- Root cause
- Component
- Contributing factor
- Detection
CirrusStreamingUpdaterFlinkJobUnstable alerts fired from 21:27; the review records the detection as human, by the engineer who started troubleshooting the next morning, and many other alerts were suppressed for maintenance.
- Remediation
- Deleted the red indices, which brought the cluster back to green
- Restored the broken indices
- Action items
- Re-enable the snapshot repositories in production CirrusSearch
- Create an operations-focused OpenSearch dashboard
- Verify or fix the Logstash pipeline for Search Platform-owned OpenSearch clusters
- Lessons
Banning nodes before removing them is a best-effort action and did not work as expected. A large operation such as decommissioning many servers at once had no maintenance plan or runbook, and the snapshots, dashboards and logging that recovery needed were not available or not documented well enough.
- Symptoms
On 18 September 2024, while a cookbook was reconfiguring Wikimedia's core MariaDB databases from one-way eqiad -> codfw replication to circular replication ahead of a datacenter switchover, replication in the codfw -> eqiad direction broke on the x1 and es6 sections. Engineers watching the run noticed at 21:28 UTC that the x1 primary was not replicating.
- Trigger
Running the cookbook sre.switchdc.databases.prepare, which set up codfw -> eqiad replication and started pt-heartbeat, on the row-based x1 and es sections.
- Root cause
- Component
- Contributing factor
- Detection
Engineers watching the first run of the automation saw on orchestrator that the x1 primary was not replicating, because monitoring of circular replication was not yet enabled. The break in the eqiad -> codfw direction was reported by Icinga.
- Remediation
- Skipped replication of the heartbeat tables with Replication_wild_ignore_table='heartbeat.%'
- Inserted the missing heartbeat row with binary logging off on the es hosts
- Inserted the missing heartbeat row with binary logging off on the x1 codfw hosts and removed the ignore rule
- Action items
- Avoid the same issue again, probably by inserting the missing row when it is detected
- Add a post-maintenance health check that replication works in both directions
- Add options to run the maintenance in another order or on each section individually
- Update the Phabricator task for each section rather than only at the end
- Fix the monitoring gaps for circular replication
- Made pt-heartbeat-wikimedia use STATEMENT format for its session on the x1 and row-based read-write es hosts, fixed in a later deploy
- Lessons
Circular replication is not a normal working mode, so the first fix only moved the break to the other side; TRUNCATE would have worked better than DELETE because it always replicates as a statement, but it was not considered because it is more dangerous. Running sql_skip_slave_counter was avoided, since it could have made things worse or lost data. The check of the replication channel should also run once all tasks are complete, not only right after setup. The combination of row-based replication, pt-heartbeat and differing heartbeat rows was not described in earlier maintenance or documentation.
- Symptoms
On 9 August 2023, right after a MediaWiki configuration deployment at 11:41 UTC, the MediaWiki deployments on Kubernetes terminated TLS with a wrong certificate, and about 1% of Wikimedia web and API traffic, around 120 requests per second, received errors for about 30 minutes. The issue was detected by a page for failed service probes on mw-api-ext and mw-api-int.
- Trigger
At 09:37 a change to the global Kubernetes defaults was merged that switched supported charts to cert-manager certificates. The next MediaWiki on Kubernetes deployment, a configuration change at 11:41, picked it up.
- Root cause
- Component
- Contributing factor
- Detection
A page from the ProbeDown alerts for mw-api-ext:4447 and mw-api-int:4446 at 11:47; a KubernetesAPILatency alert had fired at 11:46.
- Remediation
- Identified the issue and prepared a patch, https://gerrit.wikimedia.org/r/947331
- Redeployed the MediaWiki deployments on Kubernetes
- Action items
- Review whether deployment procedures and testing should be strengthened, for example against surprising changes on the next deployment and with canary deployments for Kubernetes
- Look at metrics that become unavailable or unhealthy during deployments, or at reducing the impact of deployments
- Lessons
The cause was found quickly because the person who made the breaking change was responding. The change should have been caught by CI, but the repository it was made in (hieradata) does not run the CI of the deployment charts. Only 1% of traffic going to Kubernetes kept the impact small, and the low error volume made it slower to link the errors to Kubernetes.
- Symptoms
On 23 November 2021 at 09:39, two minutes after a change on the eqiad core routers, alerts fired because Icinga in eqiad could not reach the public text-lb address in eqsin (Singapore). For about 12 minutes eqiad could not reach hosts in other data centers, such as the cache PoPs, through their public IP addresses.
- Trigger
At about 09:37 a change on cr1-eqiad and cr2-eqiad, made as part of task T295672 to influence BGP route selection, removed a BGP setting that sets the BGP "MED" attribute to the OSPF cost of reaching the BGP route's next hop.
- Root cause
- Component
- Contributing factor
- Detection
Icinga health checks against edge caching clusters alerted at 09:39; the first alert was Icinga in eqiad failing to reach the public text-lb address in eqsin.
- Remediation
- Reverted the configuration change on both eqiad routers
- Forced a reset of the BGP session between the two eqiad routers
- Action items
- Updated T295672 with further detail on the incident and a proposal to adjust the BGP sessions to address the "second IXP port" issue and prevent this from happening again
- Symptoms
On 10 September 2019 the Kubernetes API server of Wikimedia's Toolforge cluster did not come back after the maintain-kubeusers process restarted it at 18:30. To monitoring and to users the whole cluster looked down: Toolforge users could not launch tools with the webservice command on the Kubernetes backend or use kubectl, and an Icinga check against the "All k8s nodes healthy" endpoint started alerting at 18:45.
- Trigger
At 18:30 the maintain-kubeusers process restarted the Kubernetes API server, which forced it to reconnect to etcd with a CA certificate that Puppet changes the night before had left mismatched with the etcd servers.
- Root cause
- Component
- Contributing factor
- Detection
Automated monitoring: an Icinga alert on the toolschecker check http://checker.tools.wmflabs.org/k8s/nodes/ready, about "All k8s nodes healthy", fired at 18:45.
- Remediation
- Rebooted the Kubernetes master and the etcd cluster, which did not help
- Took a backup of etcd and reduced the etcd cluster to one node to rule out raft election flaps
- Brought etcd back to full capacity through a full disaster recovery from the backup
- Swapped the API server's CA certificate for the one from the etcd servers, which restored services until Puppet reverted it
- Placed the etcd Puppet CA certificate in /var/lib/puppet/client/ssl/certs/ca.pem so Puppet kept the right CA
- Action items
- Document the etcd cluster processes
- Upgrade Toolforge Kubernetes to improve etcd and Kubernetes error reporting and stop needing an API server restart to add a tool
- Make kube2proxy more resilient to API server failures
- Audit the Puppet CA certificates so the tools project has no more than one
- Make the Kubernetes control plane highly available
- Lessons
Components that often misbehave, such as etcd, drew the responders to the wrong fix, and working on etcd instead of the CA certificates made the outage worse. Puppet-managed certificates are risky, and very old Kubernetes versions log errors that are hard to read. The failure was hard to find because it surfaced at an API server restart rather than at the Puppet change, because the etcd cluster had long-standing undocumented quirks, and because no disaster recovery or node replacement procedure was written down for etcd.
These 5 cases involve BGP routing and cert-manager. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.
By root cause1
Configuration change5 cases
- Component
- Contributing factor
- Organisation
IL-0299About 60 indices of Wikimedia's CirrusSearch OpenSearch clusters went red and unwritable during a host decommissionIL-0392MariaDB replication of section x1 between eqiad and codfw broke on a missing heartbeat row during a cookbook runIL-0459MediaWiki on Kubernetes terminated TLS with a wrong certificate after a deployment, and 1% of requests failedIL-0266Eqiad routers looped traffic between each other after a BGP MED change and could not reach other sites' public IPsIL-0458Toolforge Kubernetes API server failed to start after a restart, logging "etcd cluster is unavailable or misconfigured"