IL-0512 MIT Open Learning · 2026-06-03kubernetesnetworkingaws

xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed

The non-OpenEdX xPRO application at MIT Open Learning used two domains: xpro.mit.edu, the public frontend served by Fastly, and xpro-web.odl.mit.edu, the backend origin on Kubernetes behind APISIX. cert-manager issued the origin certificate from Let's Encrypt with DNS-01 validation in Route 53. The application's Pulumi code, however, passed every Fastly service domain into the Fastly TLS subscription, so Fastly also created DNS validation records for the backend domain, including a CNAME at _acme-challenge.xpro-web.odl.mit.edu. That CNAME conflicted with the TXT record cert-manager needed for its DNS-01 challenge.

From 4 May 2026 cert-manager's renewal could not complete validation, the ACME order went stale, and the origin certificate expired at 3:29 PM EDT on 3 June. Fastly could no longer connect to the origin and the site returned 503 errors. An engineer deleted and recreated the stale cert-manager certificate resources, cert-manager issued a fresh certificate, and service was restored at 4:38 PM EDT. After the incident, PR mitodl/ol-infrastructure#4713 changed the Fastly TLS subscription to cover only xpro.mit.edu, leaving the origin certificate to cert-manager; no edxapp or xpro-openedx changes were needed.

Severity: sev1
Scope: The public xPRO site https://xpro.mit.edu and the xPRO production CMS returned 503 errors
Users affected: xPRO customers could not reliably access the production site
Region: –
Time to detect: 9m
Time to resolve: 1h 9m

Symptoms

At 3:38 PM EDT on 3 June 2026, alerts fired because both the xPRO production CMS and the main xPRO site at https://xpro.mit.edu were returning 503 errors. Fastly, in front of the site, could not connect to the Kubernetes/APISIX origin, because the origin certificate for xpro-web.odl.mit.edu had expired at 3:29 PM EDT.

In the cluster, the certificate xpro-cert had expired and its cert-manager ACME order was stale. When the cert-manager pod restarted at 4:16 PM EDT and re-evaluated the stale order, Let's Encrypt answered "404: No order for ID 507326254665" and the issuance was marked failed.

The renewal had been failing for a month. cert-manager entered the 30-day renewal window on 4 May 2026 and created xpro-cert-2 with a new CertificateRequest, Order and DNS-01 Challenge, but the challenge did not complete, Let's Encrypt deactivated the ACME authorization, and by late May or early June the order was stale. The first diagnosis stopped at a failed cert-manager renewal; why the DNS-01 challenge failed was found only in a later review.

Symptom Service outage TLS certificate errors

Root cause

Root cause Configuration change

Component cert-manager CDN and edge network Authoritative DNS servers and records

Contributing factor No configuration tests Monitoring blind spot Change with a delayed effect

Trigger: The origin certificate xpro-cert for xpro-web.odl.mit.edu, issued on 2026-03-05 with a 90-day validity period, expired at 3:29 PM EDT on 3 June 2026.

Detection

Alerts for the xPRO production CMS and the main xPRO site fired at 3:38 PM EDT on the 503 errors, and a Rootly incident was created at 4:08 PM EDT.

Remediation

Deleted and recreated the stale cert-manager certificate resources so that cert-manager issued a fresh origin certificate

Action items

Remove the backend origin domain from the non-OpenEdX xPRO Fastly TLS subscription, so Fastly manages only xpro.mit.edu
Deploy the Pulumi update for the xPRO application in Production
Trigger cert-manager renewal of the xPRO certificate after the conflicting Fastly validation CNAME is removed
Consider tests or policy checks that keep cert-manager-managed origin domains out of Fastly TLS subscriptions
Consider alerting earlier on failed cert-manager CertificateRequests and Orders, so stale ACME failures are found before the certificate expires

Lessons

Reading the outage as a failed cert-manager ACME renewal was true but incomplete; the missing part was that Fastly had been set to manage validation for a backend origin domain that belonged to cert-manager. The team concluded that TLS certificates and DNS validation records need guardrails around ownership, and that public Fastly frontend domains and Kubernetes/APISIX origin domains must not be treated as interchangeable service domains in TLS automation.

Sources

https://engineering.ol.mit.edu/runbooks_post_mortems/20260603_xpro_outage/ · runbook

Related cases

IL-0363 static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch cratesRust Project · 2023
same root cause · shared Authoritative DNS servers and records, CDN and edge network
IL-0283 Cloudflare's core proxy returned HTTP 5xx across its network after loading an oversized Bot Management feature fileCloudflare · 2025
same root cause · shared CDN and edge network
IL-0502 The JupyterHub at temple.2i2c.cloud stopped resolving in DNS after a wildcard record was added for a new cluster2i2c · 2025
same root cause · shared Authoritative DNS servers and records
IL-0339 Azure Front Door and Azure CDN users saw intermittent connection errors, timeouts and latency spikes for hoursMicrosoft Azure · 2024
same root cause · shared CDN and edge network
IL-0533 Names in chinacloudapp.cn and chinacloudsites.cn stopped resolving as DNS caches expired, breaking Azure China servicesMicrosoft Azure · 2024
same root cause · shared Authoritative DNS servers and records
IL-0254 Square's internal DNS servers failed and Square products, services and payments were unavailable for about 15 hoursSquare · 2023
same root cause · shared Authoritative DNS servers and records
IL-0459 MediaWiki on Kubernetes terminated TLS with a wrong certificate after a deployment, and 1% of requests failedWikimedia Foundation · 2023
same root cause · shared cert-manager
IL-0549 Media, attachments, Git LFS and build artifacts failed for Atlassian in us-west-1 when an old network stack was removedAtlassian · 2023
same root cause · shared Authoritative DNS servers and records

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, cert-manager, CDN and edge network, Authoritative DNS servers and records among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0512
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0512/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.