Cases2 cases

1 to 2 of 2

1. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed

   MIT Open Learning2026sev1

   Symptoms

   At 3:38 PM EDT on 3 June 2026, alerts fired because both the xPRO production CMS and the main xPRO site at https://xpro.mit.edu were returning 503 errors. Fastly, in front of the site, could not connect to the Kubernetes/APISIX origin, because the origin certificate for xpro-web.odl.mit.edu had expired at 3:29 PM EDT.

   Service outageTLS certificate errors

   Trigger

   The origin certificate xpro-cert for xpro-web.odl.mit.edu, issued on 2026-03-05 with a 90-day validity period, expired at 3:29 PM EDT on 3 June 2026.

   Root cause

   Configuration change

   Component

   cert-managerCDN and edge networkAuthoritative DNS servers and records

   Contributing factor

   No configuration testsMonitoring blind spotChange with a delayed effect

   Detection

   Alerts for the xPRO production CMS and the main xPRO site fired at 3:38 PM EDT on the 503 errors, and a Rootly incident was created at 4:08 PM EDT.

   Remediation

   • Deleted and recreated the stale cert-manager certificate resources so that cert-manager issued a fresh origin certificate

   Action items

   • Remove the backend origin domain from the non-OpenEdX xPRO Fastly TLS subscription, so Fastly manages only xpro.mit.edu
   • Deploy the Pulumi update for the xPRO application in Production
   • Trigger cert-manager renewal of the xPRO certificate after the conflicting Fastly validation CNAME is removed
   • Consider tests or policy checks that keep cert-manager-managed origin domains out of Fastly TLS subscriptions
   • Consider alerting earlier on failed cert-manager CertificateRequests and Orders, so stale ACME failures are found before the certificate expires

   Lessons

   Reading the outage as a failed cert-manager ACME renewal was true but incomplete; the missing part was that Fastly had been set to manage validation for a backend origin domain that belonged to cert-manager. The team concluded that TLS certificates and DNS validation records need guardrails around ownership, and that public Fastly frontend domains and Kubernetes/APISIX origin domains must not be treated as interchangeable service domains in TLS automation.

   View case
2. IL-0531 Azure Database for MySQL Single Server and MariaDB in West US rejected every new connection for a batch of servers

   Microsoft Azure2024

   Symptoms

   From 04:26 UTC on 19 April 2024, every new connection to a subset of Azure Database for MySQL - Single Server and Azure Database for MariaDB servers in West US failed, and retries did not help. The first report came at 04:46 UTC from Azure Databricks, which told the MySQL team that multiple databases were unreachable.

   Service outageIntermittent timeouts

   Trigger

   At 04:26 UTC, during planned maintenance in West US, the DNS update that moves new incoming connections for a batch of MySQL and MariaDB servers to a new gateway ring was made, and a procedural error pointed it at a newly built gateway ring whose configuration was incomplete.

   Root cause

   Operator error

   Component

   DatabaseAuthoritative DNS servers and records

   Contributing factor

   Monitoring blind spot

   Detection

   Service telemetry did not detect the failure because the new gateway ring had no telemetry or monitoring configured yet. At 04:46 UTC Azure Databricks reported to the MySQL team that multiple databases were unreachable.

   Remediation

   • Rolled back the DNS change and moved all connections back to the previous working gateway ring after validation

   Action items

   • Add checks, testing and sign-off procedures before switching production gateway rings
   • Add Azure Databricks alerts that detect multiple databases becoming unavailable within a short time

   View case

By root cause2