Cases11 cases

1 to 5 of 11

1. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed

   MIT Open Learning2026sev1

   Symptoms

   At 3:38 PM EDT on 3 June 2026, alerts fired because both the xPRO production CMS and the main xPRO site at https://xpro.mit.edu were returning 503 errors. Fastly, in front of the site, could not connect to the Kubernetes/APISIX origin, because the origin certificate for xpro-web.odl.mit.edu had expired at 3:29 PM EDT.

   Service outageTLS certificate errors

   Trigger

   The origin certificate xpro-cert for xpro-web.odl.mit.edu, issued on 2026-03-05 with a 90-day validity period, expired at 3:29 PM EDT on 3 June 2026.

   Root cause

   Configuration change

   Component

   cert-managerCDN and edge networkAuthoritative DNS servers and records

   Contributing factor

   No configuration testsMonitoring blind spotChange with a delayed effect

   Detection

   Alerts for the xPRO production CMS and the main xPRO site fired at 3:38 PM EDT on the 503 errors, and a Rootly incident was created at 4:08 PM EDT.

   Remediation

   • Deleted and recreated the stale cert-manager certificate resources so that cert-manager issued a fresh origin certificate

   Action items

   • Remove the backend origin domain from the non-OpenEdX xPRO Fastly TLS subscription, so Fastly manages only xpro.mit.edu
   • Deploy the Pulumi update for the xPRO application in Production
   • Trigger cert-manager renewal of the xPRO certificate after the conflicting Fastly validation CNAME is removed
   • Consider tests or policy checks that keep cert-manager-managed origin domains out of Fastly TLS subscriptions
   • Consider alerting earlier on failed cert-manager CertificateRequests and Orders, so stale ACME failures are found before the certificate expires

   Lessons

   Reading the outage as a failed cert-manager ACME renewal was true but incomplete; the missing part was that Fastly had been set to manage validation for a backend origin domain that belonged to cert-manager. The team concluded that TLS certificates and DNS validation records need guardrails around ownership, and that public Fastly frontend domains and Kubernetes/APISIX origin domains must not be treated as interchangeable service domains in TLS automation.

   View case
2. IL-0226 Azure Front Door edge servers crashed worldwide on incompatible configuration metadata, causing timeouts and DNS errors

   Microsoft Azure2025

   Symptoms

   From 15:41 UTC on 29 October 2025, customers and Microsoft services using Azure Front Door (AFD) and Azure Content Delivery Network (CDN) got connection timeout errors and DNS resolution issues. Monitoring alerts started the investigation at 15:48 UTC.

   Intermittent timeoutsDNS resolution failuresService outageHigh latency

   Trigger

   At 15:35 UTC a specific sequence of valid customer configuration changes, made across two different AFD control plane build versions, produced incompatible customer configuration metadata that was deployed to the edge sites.

   Root cause

   Software defect

   Component

   CDN and edge networkAuthoritative DNS servers and records

   Contributing factor

   Gap in automated testsChange with a delayed effectIneffective rollbackExcessive blast radius

   Detection

   The configuration protection system detected the data plane impact and blocked configuration changes at 15:43 UTC, and monitoring alerts started the investigation at 15:48 UTC.

   Remediation

   • The configuration protection system automatically blocked all new and in-flight configuration changes worldwide
   • Removed the problematic customer configurations from the latest LKG snapshot by hand
   • Blocked all customer configuration changes from propagating to the data plane
   • Deployed the updated LKG configuration across the global fleet
   • Moved Azure Portal away from Azure Front Door
   • Rebalanced traffic by hand to a few healthy edge sites, then switched back to automatic traffic management
   • Blocked all AFD customer configuration changes at the Azure Resource Manager level until safeguards were in place

   Action items

   • Fix the control plane incompatibility and the data plane bug
   • Process each customer configuration completely and synchronously before production stages
   • Add stages to the phased configuration rollout, with longer bake time between them
   • Move configuration processing on data plane servers into isolated worker processes
   • Reduce configuration propagation time from 45 minutes to about 15 minutes
   • Test backwards compatibility with configurations generated by earlier control plane builds
   • Move critical first-party infrastructure to an active-active setup with fail away
   • Isolate customer configuration and traffic with micro cell segmentation of the AFD data plane
   • Restore customer configurations within one hour by using the local configuration cache
   • Cut data plane recovery further, to about 10 minutes
   • Remove delays in Azure Service Health alerts by raising resource thresholds
   • Extend automated Azure Service Health alerts to similar classes of service degradation
   • Fail over support request systems faster so Premier and Unified customers can always open requests

   Lessons

   Microsoft describes configuration propagation risk as inherent to a global Content Delivery Network, in which route, WAF and origin changes must be deployed quickly worldwide. Both this incident and the 9 October AFD incident came from that risk. This time the incompatible metadata passed a protection system that advances only on healthy data plane signals, because the crash came later from delayed asynchronous processing. Some learnings and repair items from the earlier incident applied here as well. Microsoft advises customers to build a resilient global HTTP ingress layer that stays available during regional or network disruptions, to manage resources through the REST API or PowerShell when management portals are unavailable, and to configure Azure Service Health alerts so the right people hear about service issues.

   View case
3. IL-0502 The JupyterHub at temple.2i2c.cloud stopped resolving in DNS after a wildcard record was added for a new cluster

   2i2c2025

   Symptoms

   On 20 August 2025 at 10:02 AM Pacific time, an automated uptime check on temple.2i2c.cloud started failing and paged 2i2c: the JupyterHub at temple.2i2c.cloud could not be reached, shortly before the community that uses it was to give a demo.

   DNS resolution failuresService outage

   Trigger

   At 9:21 AM, while setting up a new cluster for Temple, a wildcard DNS record *.temple.2i2c.cloud was added pointing to the new cluster's Ingress IP.

   Root cause

   Configuration change

   Component

   Authoritative DNS servers and records

   Contributing factor

   Unverified assumptionCache-masked failureUnreviewed defaults

   Detection

   Automated uptime alerts caught the failure at 10:02 AM, before the community reported it in a support ticket at 10:09 AM.

   Remediation

   • Added an explicit A record for temple.2i2c.cloud in Namecheap
   • Lowered the DNS TTL from the 30-minute default to 5 minutes
   • Offered the community member running the demo an /etc/hosts entry as a workaround

   Action items

   • Give everyone on the team access to Namecheap, or move to another DNS provider
   • Use a 5-minute TTL instead of 30 minutes for all new DNS entries
   • Document that adding a *.X.2i2c.cloud DNS entry requires adding an X.2i2c.cloud entry as well

   Lessons

   Adding a wildcard record for a subdomain stopped a broader wildcard from answering for that subdomain's own name, so a *.X.2i2c.cloud record needs an X.2i2c.cloud record alongside it. The 30-minute default TTL at Namecheap meant any DNS fix could take that long to propagate, and the engineer creating the original record had no access to the DNS provider.

   View case
4. IL-0260 zoom.us stopped resolving for about two hours after GoDaddy Registry placed a server block on the domain

   Zoom2025

   Symptoms

   On 16 April 2025 customers in the US and International regions could not reach Zoom services because lookups of the zoom.us domain failed. Starting, joining or scheduling a meeting needs a DNS lookup, so none of those actions completed; people already in a Zoom meeting or on a Zoom Phone call were not affected.

   DNS resolution failuresService outage

   Trigger

   A communication error between Zoom's domain registrar, Markmonitor, and GoDaddy Registry led GoDaddy Registry to place a server block on zoom.us by mistake, which shut the domain down.

   Root cause

   Dependency failure

   Component

   Authoritative DNS servers and records

   Contributing factor

   No safeguard on destructive change

   Remediation

   • Identified and removed the server block on zoom.us together with Markmonitor and GoDaddy
   • Asked users who still had connection problems to flush their DNS cache and reconnect

   Action items

   • Put a registry lock on zoom.us that restricts server block commands on the domain
   • Implement additional technical changes and processes with GoDaddy Registry and Markmonitor so the issue does not recur

   View case
5. IL-0348 Scaleway's FR-PAR region lost public internet connectivity when one 100 Gbps backbone link was saturated

   Scaleway2025

   Symptoms

   On 10 March 2025 at 16:22 UTC, public internet connectivity in Scaleway's FR-PAR region broke for all products. Most internet traffic between the FR-PAR-2 availability zone and the rest of the PAR region was blackholed for about five minutes, while VPC traffic and Dedibox RPN kept working.

   Packet lossService outageHigh latency

   Trigger

   While Scaleway's network team connected a new data center to the FR-PAR backbone, a human mistake in a router configuration rerouted more than 1 Tbps of traffic between OpCore DC3 and DC5 through a single 100 Gbps link.

   Root cause

   Operator error

   Component

   Backbone network (WAN)Object storageAuthoritative DNS servers and records

   Contributing factor

   Automation amplification

   Detection

   The router configuration error was detected quickly and rolled back within four minutes.

   Remediation

   • Rolled back the router configuration, restoring the correct configuration at 16:27 UTC

   Action items

   • Harden the peer review and production communication process for changes
   • Add failsafes and tests before network configuration is applied at backbone level
   • Deploy the fix for the Object Storage synchronisation software issue

   View case

By root cause5