Incident Library
Applied filters Root causeConfiguration changeFactorChange with a delayed effectOrganisationMIT Open LearningClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed
    MIT Open Learning2026sev1
    Symptoms

    At 3:38 PM EDT on 3 June 2026, alerts fired because both the xPRO production CMS and the main xPRO site at https://xpro.mit.edu were returning 503 errors. Fastly, in front of the site, could not connect to the Kubernetes/APISIX origin, because the origin certificate for xpro-web.odl.mit.edu had expired at 3:29 PM EDT.

    Service outageTLS certificate errors
    Trigger

    The origin certificate xpro-cert for xpro-web.odl.mit.edu, issued on 2026-03-05 with a 90-day validity period, expired at 3:29 PM EDT on 3 June 2026.

    Root cause
    Configuration change
    Component
    cert-managerCDN and edge networkAuthoritative DNS servers and records
    Contributing factor
    No configuration testsMonitoring blind spotChange with a delayed effect
    Detection

    Alerts for the xPRO production CMS and the main xPRO site fired at 3:38 PM EDT on the 503 errors, and a Rootly incident was created at 4:08 PM EDT.

    Remediation
    • Deleted and recreated the stale cert-manager certificate resources so that cert-manager issued a fresh origin certificate
    Action items
    • Remove the backend origin domain from the non-OpenEdX xPRO Fastly TLS subscription, so Fastly manages only xpro.mit.edu
    • Deploy the Pulumi update for the xPRO application in Production
    • Trigger cert-manager renewal of the xPRO certificate after the conflicting Fastly validation CNAME is removed
    • Consider tests or policy checks that keep cert-manager-managed origin domains out of Fastly TLS subscriptions
    • Consider alerting earlier on failed cert-manager CertificateRequests and Orders, so stale ACME failures are found before the certificate expires
    Lessons

    Reading the outage as a failed cert-manager ACME renewal was true but incomplete; the missing part was that Fastly had been set to manage validation for a backend origin domain that belonged to cert-manager. The team concluded that TLS certificates and DNS validation records need guardrails around ownership, and that public Fastly frontend domains and Kubernetes/APISIX origin domains must not be treated as interchangeable service domains in TLS automation.

    View case
  2. IL-0511 MIT xPro stopped responding after a configuration deploy as pgbouncer logins to RDS failed with 'wrong password type'
    MIT Open Learning2023
    Symptoms

    On 30 October 2023 at 3:51PM, six minutes after a configuration update was deployed to the xPro production Heroku stack, Pingdom alerted the DevOps on-call engineer, who confirmed at 3:52PM that the site was not responding.

    Service outageCredentials unavailable
    Trigger

    A configuration update applied through Salt at 3:45PM found that the Vault lease of the database credentials was about to expire and could not be renewed, so Vault generated new credentials.

    Root cause
    Configuration change
    Component
    DatabaseSecrets store
    Contributing factor
    Unreviewed defaultsChange with a delayed effectConfiguration not in code
    Detection

    Pingdom alerted the DevOps on-call engineer at 3:51PM, and the alert went through OpsGenie; Pingdom closed it automatically at 4:25PM.

    Remediation
    • Created a new database parameter group and applied it to the running RDS instance
    • Cleared the cached credentials and re-applied the Heroku configuration through Salt
    Action items
    • Move the management of xPro resources, including the RDS instance, into Pulumi, where AWS default parameter groups are not used
    Lessons

    Almost none of the configuration for this stack, the RDS instance in particular, was managed as code, and that was one of the main reasons the fix took long. Under Pulumi the instance would not have used an AWS default parameter group, and the fix would have been a change to an attached group.

    View case
Check your cluster for this

These 2 cases involve CDN and edge network and cert-manager. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change2 cases

Component
CDN and edge network1cert-manager1Database1Authoritative DNS servers and records1Secrets store1
Contributing factor
Change with a delayed effect2Configuration not in code1Unreviewed defaults1Monitoring blind spot1No configuration tests1
Organisation
MIT Open Learning2
  1. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed
    MIT Open Learning2026
  2. IL-0511 MIT xPro stopped responding after a configuration deploy as pgbouncer logins to RDS failed with 'wrong password type'
    MIT Open Learning2023