Cases6 cases

6 to 6 of 6

1. IL-0459 MediaWiki on Kubernetes terminated TLS with a wrong certificate after a deployment, and 1% of requests failed

   Wikimedia Foundation2023

   Symptoms

   On 9 August 2023, right after a MediaWiki configuration deployment at 11:41 UTC, the MediaWiki deployments on Kubernetes terminated TLS with a wrong certificate, and about 1% of Wikimedia web and API traffic, around 120 requests per second, received errors for about 30 minutes. The issue was detected by a page for failed service probes on mw-api-ext and mw-api-int.

   High error rateTLS certificate errorsControl plane lag

   Trigger

   At 09:37 a change to the global Kubernetes defaults was merged that switched supported charts to cert-manager certificates. The next MediaWiki on Kubernetes deployment, a configuration change at 11:41, picked it up.

   Root cause

   Configuration change

   Component

   cert-managerHelm

   Contributing factor

   Change with a delayed effectNo configuration tests

   Detection

   A page from the ProbeDown alerts for mw-api-ext:4447 and mw-api-int:4446 at 11:47; a KubernetesAPILatency alert had fired at 11:46.

   Remediation

   • Identified the issue and prepared a patch, https://gerrit.wikimedia.org/r/947331
   • Redeployed the MediaWiki deployments on Kubernetes

   Action items

   • Review whether deployment procedures and testing should be strengthened, for example against surprising changes on the next deployment and with canary deployments for Kubernetes
   • Look at metrics that become unavailable or unhealthy during deployments, or at reducing the impact of deployments

   Lessons

   The cause was found quickly because the person who made the breaking change was responding. The change should have been caught by CI, but the repository it was made in (hieradata) does not run the CI of the deployment charts. Only 1% of traffic going to Kubernetes kept the impact small, and the low error volume made it slower to link the errors to Kubernetes.

   View case

By root cause5