IL-0459 Wikimedia Foundation · 2023-08-09kubernetes

MediaWiki on Kubernetes terminated TLS with a wrong certificate after a deployment, and 1% of requests failed

A change to Wikimedia's global Kubernetes defaults, merged at 09:37 UTC on 9 August 2023, switched the charts that support it to cert-manager certificates. Nothing changed until the next MediaWiki on Kubernetes deployment, a MediaWiki configuration change at 11:41, after which the MediaWiki deployments used a wrong certificate for TLS termination. At the time only 1% of traffic was routed to Kubernetes, so about 1% of requests, around 120 per second, failed.

Probe pages fired at 11:47. The person who had made the breaking change was one of the two responders, so the cause was found quickly: a patch was prepared at 11:58, the MediaWiki deployments were redeployed from 12:08, and the outage ended at 12:16. The write-up notes that the root cause should have been caught in CI, but the change was made in the hieradata repository, whose CI does not cover the deployment charts.

Severity: –
Scope: MediaWiki web and API availability: about 1% of traffic (120 requests/s) received errors for about 30 minutes
Users affected: –
Region: –
Time to detect: 6m
Time to resolve: 35m

Symptoms

On 9 August 2023, right after a MediaWiki configuration deployment at 11:41 UTC, the MediaWiki deployments on Kubernetes terminated TLS with a wrong certificate, and about 1% of Wikimedia web and API traffic, around 120 requests per second, received errors for about 30 minutes. The issue was detected by a page for failed service probes on mw-api-ext and mw-api-int.

The alerts read "(ProbeDown) firing: (2) Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) #page" and "(ProbeDown) firing: Service mw-api-int:4446 has failed probes (http_mw-api-int_ip4)". One minute earlier, at 11:46, "(KubernetesAPILatency) firing: High Kubernetes API latency (POST pods) on k8s@codfw" had fired, and at 12:02 Icinga reported the httpbb_kubernetes_mw-api-int_hourly check on cumin2002 as CRITICAL.

Because only a small share of requests failed, it took some time to tie the errors to Kubernetes. Around 11:56 a responder suspected Kubernetes, and at 11:58 the cause was identified.

Symptom High error rate TLS certificate errors Control plane lag

Root cause

Root cause Configuration change

Component cert-manager Helm

Contributing factor Change with a delayed effect No configuration tests

Trigger: At 09:37 a change to the global Kubernetes defaults was merged that switched supported charts to cert-manager certificates. The next MediaWiki on Kubernetes deployment, a configuration change at 11:41, picked it up.

Detection

A page from the ProbeDown alerts for mw-api-ext:4447 and mw-api-int:4446 at 11:47; a KubernetesAPILatency alert had fired at 11:46.

Remediation

Identified the issue and prepared a patch, https://gerrit.wikimedia.org/r/947331
Redeployed the MediaWiki deployments on Kubernetes

Action items

Review whether deployment procedures and testing should be strengthened, for example against surprising changes on the next deployment and with canary deployments for Kubernetes
Look at metrics that become unavailable or unhealthy during deployments, or at reducing the impact of deployments

Lessons

The cause was found quickly because the person who made the breaking change was responding. The change should have been caught by CI, but the repository it was made in (hieradata) does not run the CI of the deployment charts. Only 1% of traffic going to Kubernetes kept the impact small, and the low error volume made it slower to link the errors to Kubernetes.

Sources

https://wikitech.wikimedia.org/wiki/Incidents/2023-08-09_mw-on-k8s_outage_due_to_wrong_tls_cert · runbook · archived copy

References

https://gerrit.wikimedia.org/r/947331

Related cases

IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failedMIT Open Learning · 2026 · sev1
same root cause · shared cert-manager
IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errorsGitLab · 2022
same root cause · shared Helm

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, cert-manager, Helm among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0459
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0459/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.