Incident Library
Applied filters Componentcert-managerFactorChange with a delayed effectDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases4 cases

1 to 4 of 4

  1. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed
    MIT Open Learning2026sev1
    Symptoms

    At 3:38 PM EDT on 3 June 2026, alerts fired because both the xPRO production CMS and the main xPRO site at https://xpro.mit.edu were returning 503 errors. Fastly, in front of the site, could not connect to the Kubernetes/APISIX origin, because the origin certificate for xpro-web.odl.mit.edu had expired at 3:29 PM EDT.

    Service outageTLS certificate errors
    Trigger

    The origin certificate xpro-cert for xpro-web.odl.mit.edu, issued on 2026-03-05 with a 90-day validity period, expired at 3:29 PM EDT on 3 June 2026.

    Root cause
    Configuration change
    Component
    cert-managerCDN and edge networkAuthoritative DNS servers and records
    Contributing factor
    No configuration testsMonitoring blind spotChange with a delayed effect
    Detection

    Alerts for the xPRO production CMS and the main xPRO site fired at 3:38 PM EDT on the 503 errors, and a Rootly incident was created at 4:08 PM EDT.

    Remediation
    • Deleted and recreated the stale cert-manager certificate resources so that cert-manager issued a fresh origin certificate
    Action items
    • Remove the backend origin domain from the non-OpenEdX xPRO Fastly TLS subscription, so Fastly manages only xpro.mit.edu
    • Deploy the Pulumi update for the xPRO application in Production
    • Trigger cert-manager renewal of the xPRO certificate after the conflicting Fastly validation CNAME is removed
    • Consider tests or policy checks that keep cert-manager-managed origin domains out of Fastly TLS subscriptions
    • Consider alerting earlier on failed cert-manager CertificateRequests and Orders, so stale ACME failures are found before the certificate expires
    Lessons

    Reading the outage as a failed cert-manager ACME renewal was true but incomplete; the missing part was that Fastly had been set to manage validation for a backend origin domain that belonged to cert-manager. The team concluded that TLS certificates and DNS validation records need guardrails around ownership, and that public Fastly frontend domains and Kubernetes/APISIX origin domains must not be treated as interchangeable service domains in TLS automation.

    View case
  2. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates
    2i2c2026
    Symptoms

    An attempt to launch a new hub failed because its ingress could not obtain a new TLS certificate. The day before, the same engineer had recreated the support deployment on the 2i2c-aws-us cluster and seen its TLS certificate orders stuck pending.

    Misrouted requestsDeploys failing
    Trigger

    Creating new hub deployments that needed new TLS certificates on clusters recently migrated from ingress-nginx to nginx-ingress.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllercert-manager
    Contributing factor
    Change with a delayed effect
    Detection

    An attempt to launch a new hub failed; the engineer had seen certificate orders stuck pending on another cluster the day before and declared an incident.

    Remediation
    • annotated the Ingress objects that need TLS so that cert-manager could edit them in place to complete the ACME challenge
    Action items
    • Follow-up tracked in https://github.com/2i2c-org/infrastructure/issues/8016
    • Follow-up tracked in https://github.com/2i2c-org/infrastructure/issues/3278
    Lessons

    Swapping one ingress controller for another can change how Ingress objects that share a host are combined. cert-manager's temporary challenge Ingress depends on that behaviour, so the breakage shows up when a deployment needs a new certificate. 2i2c notes that it got lucky: it had already learned of this possible cause while testing on an internal cluster, and changing the cert-manager configuration was quick.

    View case
  3. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down
    PortSwigger2026
    Symptoms

    From 10:00 UTC on 11 January 2026 every BSEE SaaS instance in PortSwigger's US and Canada regions was unavailable: customers could not open the Web UI or start scans. The platform team learned of it only when customers reported it through support at 09:52 UTC the next day.

    Service outageTLS certificate errors
    Trigger

    The upgrade of the AWS Load Balancer Controller from v1.14.1 to v1.16.0 on 28 November 2025 left an orphaned webhook TLS secret that cert-manager could not renew, and the certificate in it expired on 8 January 2026.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllerAdmission webhookcert-managerHelmConfigMaps and SecretsCloud load balancer
    Contributing factor
    Change with a delayed effectMonitoring blind spot
    Detection

    Customers reported the outage through support at 09:52 UTC on 12 January 2026, almost 24 hours after impact began, and the platform team began investigating at 10:00 UTC.

    Remediation
    • Deleted the orphaned aws-load-balancer-tls secrets in the US and Canada clusters
    • Redeployed the AWS Load Balancer Controller configuration with Helm
    • Verified that new certificates were generated and the webhooks worked again
    Action items
    • Audit all clusters for orphaned secrets that have no owning Certificate resource and resolve them
    • Add high-level application access alerting for each SaaS cluster from an external probe
    View case
  4. IL-0459 MediaWiki on Kubernetes terminated TLS with a wrong certificate after a deployment, and 1% of requests failed
    Wikimedia Foundation2023
    Symptoms

    On 9 August 2023, right after a MediaWiki configuration deployment at 11:41 UTC, the MediaWiki deployments on Kubernetes terminated TLS with a wrong certificate, and about 1% of Wikimedia web and API traffic, around 120 requests per second, received errors for about 30 minutes. The issue was detected by a page for failed service probes on mw-api-ext and mw-api-int.

    High error rateTLS certificate errorsControl plane lag
    Trigger

    At 09:37 a change to the global Kubernetes defaults was merged that switched supported charts to cert-manager certificates. The next MediaWiki on Kubernetes deployment, a configuration change at 11:41, picked it up.

    Root cause
    Configuration change
    Component
    cert-managerHelm
    Contributing factor
    Change with a delayed effectNo configuration tests
    Detection

    A page from the ProbeDown alerts for mw-api-ext:4447 and mw-api-int:4446 at 11:47; a KubernetesAPILatency alert had fired at 11:46.

    Remediation
    • Identified the issue and prepared a patch, https://gerrit.wikimedia.org/r/947331
    • Redeployed the MediaWiki deployments on Kubernetes
    Action items
    • Review whether deployment procedures and testing should be strengthened, for example against surprising changes on the next deployment and with canary deployments for Kubernetes
    • Look at metrics that become unavailable or unhealthy during deployments, or at reducing the impact of deployments
    Lessons

    The cause was found quickly because the person who made the breaking change was responding. The change should have been caught by CI, but the repository it was made in (hieradata) does not run the CI of the deployment charts. Only 1% of traffic going to Kubernetes kept the impact small, and the low error volume made it slower to link the errors to Kubernetes.

    View case
Check your cluster for this

These 4 cases involve cert-manager and Helm. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Configuration change2 cases

Component
cert-manager2CDN and edge network1Authoritative DNS servers and records1Helm1
Contributing factor
Change with a delayed effect2No configuration tests2Monitoring blind spot1
Organisation
MIT Open Learning1Wikimedia Foundation1
  1. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed
    MIT Open Learning2026
  2. IL-0459 MediaWiki on Kubernetes terminated TLS with a wrong certificate after a deployment, and 1% of requests failed
    Wikimedia Foundation2023

Breaking change in an upgrade2 cases

Component
cert-manager2Ingress controller2Admission webhook1Cloud load balancer1ConfigMaps and Secrets1Helm1
Contributing factor
Change with a delayed effect2Monitoring blind spot1
Organisation
2i2c1PortSwigger1
  1. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates
    2i2c2026
  2. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down
    PortSwigger2026