Incident Library
Applied filters Root causeConfiguration changeComponentHelmOrganisationWikimedia FoundationClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0459 MediaWiki on Kubernetes terminated TLS with a wrong certificate after a deployment, and 1% of requests failed
    Wikimedia Foundation2023
    Symptoms

    On 9 August 2023, right after a MediaWiki configuration deployment at 11:41 UTC, the MediaWiki deployments on Kubernetes terminated TLS with a wrong certificate, and about 1% of Wikimedia web and API traffic, around 120 requests per second, received errors for about 30 minutes. The issue was detected by a page for failed service probes on mw-api-ext and mw-api-int.

    High error rateTLS certificate errorsControl plane lag
    Trigger

    At 09:37 a change to the global Kubernetes defaults was merged that switched supported charts to cert-manager certificates. The next MediaWiki on Kubernetes deployment, a configuration change at 11:41, picked it up.

    Root cause
    Configuration change
    Component
    cert-managerHelm
    Contributing factor
    Change with a delayed effectNo configuration tests
    Detection

    A page from the ProbeDown alerts for mw-api-ext:4447 and mw-api-int:4446 at 11:47; a KubernetesAPILatency alert had fired at 11:46.

    Remediation
    • Identified the issue and prepared a patch, https://gerrit.wikimedia.org/r/947331
    • Redeployed the MediaWiki deployments on Kubernetes
    Action items
    • Review whether deployment procedures and testing should be strengthened, for example against surprising changes on the next deployment and with canary deployments for Kubernetes
    • Look at metrics that become unavailable or unhealthy during deployments, or at reducing the impact of deployments
    Lessons

    The cause was found quickly because the person who made the breaking change was responding. The change should have been caught by CI, but the repository it was made in (hieradata) does not run the CI of the deployment charts. Only 1% of traffic going to Kubernetes kept the impact small, and the low error volume made it slower to link the errors to Kubernetes.

    View case
Check your cluster for this

This case involves cert-manager and Helm. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
cert-managerHelm
Contributing factor
Change with a delayed effectNo configuration tests
Organisation
Wikimedia Foundation
  1. IL-0459 MediaWiki on Kubernetes terminated TLS with a wrong certificate after a deployment, and 1% of requests failed
    Wikimedia Foundation2023