Graph
Cases3 cases
1 to 3 of 3
- Symptoms
At 3:38 PM EDT on 3 June 2026, alerts fired because both the xPRO production CMS and the main xPRO site at https://xpro.mit.edu were returning 503 errors. Fastly, in front of the site, could not connect to the Kubernetes/APISIX origin, because the origin certificate for xpro-web.odl.mit.edu had expired at 3:29 PM EDT.
- Trigger
The origin certificate xpro-cert for xpro-web.odl.mit.edu, issued on 2026-03-05 with a 90-day validity period, expired at 3:29 PM EDT on 3 June 2026.
- Root cause
- Component
- Contributing factor
- Detection
Alerts for the xPRO production CMS and the main xPRO site fired at 3:38 PM EDT on the 503 errors, and a Rootly incident was created at 4:08 PM EDT.
- Remediation
- Deleted and recreated the stale cert-manager certificate resources so that cert-manager issued a fresh origin certificate
- Action items
- Remove the backend origin domain from the non-OpenEdX xPRO Fastly TLS subscription, so Fastly manages only xpro.mit.edu
- Deploy the Pulumi update for the xPRO application in Production
- Trigger cert-manager renewal of the xPRO certificate after the conflicting Fastly validation CNAME is removed
- Consider tests or policy checks that keep cert-manager-managed origin domains out of Fastly TLS subscriptions
- Consider alerting earlier on failed cert-manager CertificateRequests and Orders, so stale ACME failures are found before the certificate expires
- Lessons
Reading the outage as a failed cert-manager ACME renewal was true but incomplete; the missing part was that Fastly had been set to manage validation for a backend origin domain that belonged to cert-manager. The team concluded that TLS certificates and DNS validation records need guardrails around ownership, and that public Fastly frontend domains and Kubernetes/APISIX origin domains must not be treated as interchangeable service domains in TLS automation.
- Symptoms
On 9 August 2023, right after a MediaWiki configuration deployment at 11:41 UTC, the MediaWiki deployments on Kubernetes terminated TLS with a wrong certificate, and about 1% of Wikimedia web and API traffic, around 120 requests per second, received errors for about 30 minutes. The issue was detected by a page for failed service probes on mw-api-ext and mw-api-int.
- Trigger
At 09:37 a change to the global Kubernetes defaults was merged that switched supported charts to cert-manager certificates. The next MediaWiki on Kubernetes deployment, a configuration change at 11:41, picked it up.
- Root cause
- Component
- Contributing factor
- Detection
A page from the ProbeDown alerts for mw-api-ext:4447 and mw-api-int:4446 at 11:47; a KubernetesAPILatency alert had fired at 11:46.
- Remediation
- Identified the issue and prepared a patch, https://gerrit.wikimedia.org/r/947331
- Redeployed the MediaWiki deployments on Kubernetes
- Action items
- Review whether deployment procedures and testing should be strengthened, for example against surprising changes on the next deployment and with canary deployments for Kubernetes
- Look at metrics that become unavailable or unhealthy during deployments, or at reducing the impact of deployments
- Lessons
The cause was found quickly because the person who made the breaking change was responding. The change should have been caught by CI, but the repository it was made in (hieradata) does not run the CI of the deployment charts. Only 1% of traffic going to Kubernetes kept the impact small, and the low error volume made it slower to link the errors to Kubernetes.
- Symptoms
On 10 September 2019 the Kubernetes API server of Wikimedia's Toolforge cluster did not come back after the maintain-kubeusers process restarted it at 18:30. To monitoring and to users the whole cluster looked down: Toolforge users could not launch tools with the webservice command on the Kubernetes backend or use kubectl, and an Icinga check against the "All k8s nodes healthy" endpoint started alerting at 18:45.
- Trigger
At 18:30 the maintain-kubeusers process restarted the Kubernetes API server, which forced it to reconnect to etcd with a CA certificate that Puppet changes the night before had left mismatched with the etcd servers.
- Root cause
- Component
- Contributing factor
- Detection
Automated monitoring: an Icinga alert on the toolschecker check http://checker.tools.wmflabs.org/k8s/nodes/ready, about "All k8s nodes healthy", fired at 18:45.
- Remediation
- Rebooted the Kubernetes master and the etcd cluster, which did not help
- Took a backup of etcd and reduced the etcd cluster to one node to rule out raft election flaps
- Brought etcd back to full capacity through a full disaster recovery from the backup
- Swapped the API server's CA certificate for the one from the etcd servers, which restored services until Puppet reverted it
- Placed the etcd Puppet CA certificate in /var/lib/puppet/client/ssl/certs/ca.pem so Puppet kept the right CA
- Action items
- Document the etcd cluster processes
- Upgrade Toolforge Kubernetes to improve etcd and Kubernetes error reporting and stop needing an API server restart to add a tool
- Make kube2proxy more resilient to API server failures
- Audit the Puppet CA certificates so the tools project has no more than one
- Make the Kubernetes control plane highly available
- Lessons
Components that often misbehave, such as etcd, drew the responders to the wrong fix, and working on etcd instead of the CA certificates made the outage worse. Puppet-managed certificates are risky, and very old Kubernetes versions log errors that are hard to read. The failure was hard to find because it surfaced at an API server restart rather than at the Puppet change, because the etcd cluster had long-standing undocumented quirks, and because no disaster recovery or node replacement procedure was written down for etcd.
These 3 cases involve cert-manager and CDN and edge network. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.
By root cause1
Configuration change3 cases
- Component
- Contributing factor
- Organisation
IL-0512xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failedIL-0459MediaWiki on Kubernetes terminated TLS with a wrong certificate after a deployment, and 1% of requests failedIL-0458Toolforge Kubernetes API server failed to start after a restart, logging "etcd cluster is unavailable or misconfigured"