Cases2 cases

1 to 2 of 2

1. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed

   MIT Open Learning2026sev1

   Symptoms

   At 3:38 PM EDT on 3 June 2026, alerts fired because both the xPRO production CMS and the main xPRO site at https://xpro.mit.edu were returning 503 errors. Fastly, in front of the site, could not connect to the Kubernetes/APISIX origin, because the origin certificate for xpro-web.odl.mit.edu had expired at 3:29 PM EDT.

   Service outageTLS certificate errors

   Trigger

   The origin certificate xpro-cert for xpro-web.odl.mit.edu, issued on 2026-03-05 with a 90-day validity period, expired at 3:29 PM EDT on 3 June 2026.

   Root cause

   Configuration change

   Component

   cert-managerCDN and edge networkAuthoritative DNS servers and records

   Contributing factor

   No configuration testsMonitoring blind spotChange with a delayed effect

   Detection

   Alerts for the xPRO production CMS and the main xPRO site fired at 3:38 PM EDT on the 503 errors, and a Rootly incident was created at 4:08 PM EDT.

   Remediation

   • Deleted and recreated the stale cert-manager certificate resources so that cert-manager issued a fresh origin certificate

   Action items

   • Remove the backend origin domain from the non-OpenEdX xPRO Fastly TLS subscription, so Fastly manages only xpro.mit.edu
   • Deploy the Pulumi update for the xPRO application in Production
   • Trigger cert-manager renewal of the xPRO certificate after the conflicting Fastly validation CNAME is removed
   • Consider tests or policy checks that keep cert-manager-managed origin domains out of Fastly TLS subscriptions
   • Consider alerting earlier on failed cert-manager CertificateRequests and Orders, so stale ACME failures are found before the certificate expires

   Lessons

   Reading the outage as a failed cert-manager ACME renewal was true but incomplete; the missing part was that Fastly had been set to manage validation for a backend origin domain that belonged to cert-manager. The team concluded that TLS certificates and DNS validation records need guardrails around ownership, and that public Fastly frontend domains and Kubernetes/APISIX origin domains must not be treated as interchangeable service domains in TLS automation.

   View case
2. IL-0283 Cloudflare's core proxy returned HTTP 5xx across its network after loading an oversized Bot Management feature file

   Cloudflare2025

   Symptoms

   From about 11:20 UTC on 18 November 2025, Cloudflare's network started failing to deliver core traffic, and Internet users visiting Cloudflare customers' sites got an error page that pointed to a failure inside Cloudflare's network. The volume of HTTP 5xx responses served by the network jumped far above its normal low baseline.

   High error rateService outageHigh latencyResponse quality degradation

   Trigger

   At 11:05 UTC Cloudflare deployed a ClickHouse access control change that made users' existing implicit access to the underlying tables in the r0 database explicit, so the metadata of those tables became visible in system tables such as system.columns. The change was being rolled out gradually across the cluster.

   Root cause

   Configuration change

   Component

   DatabaseCDN and edge network

   Contributing factor

   Unverified assumptionPartial rolloutNo staged rolloutNo configuration testsOverlapping failuresRetry storm

   Detection

   The first automated test detected the issue at 11:31 UTC, three minutes after the first errors on customer HTTP traffic at 11:28. Manual investigation started at 11:32 and the incident call was created at 11:35.

   Remediation

   • Bypassed the core proxy for Workers KV and Access so they fell back to a prior proxy version, which reduced their impact
   • Stopped the automatic creation and deployment of new Bot Management feature files
   • Inserted a known good feature file into the distribution queue and deployed it globally
   • Forced a restart of the core proxy
   • Scaled control plane concurrency to clear the backlog of dashboard logins
   • Restarted the remaining services that had entered a bad state

   Action items

   • Harden the ingestion of Cloudflare-generated configuration files the same way as user-generated input
   • Enable more global kill switches for features
   • Stop core dumps and other error reports from being able to overwhelm system resources
   • Review the failure modes for error conditions across all core proxy modules

   View case

By root cause1