Incident Library
Applied filters Root causeConfiguration changeComponentAuthoritative DNS servers and recordsSymptomService outageClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed
    MIT Open Learning2026sev1
    Symptoms

    At 3:38 PM EDT on 3 June 2026, alerts fired because both the xPRO production CMS and the main xPRO site at https://xpro.mit.edu were returning 503 errors. Fastly, in front of the site, could not connect to the Kubernetes/APISIX origin, because the origin certificate for xpro-web.odl.mit.edu had expired at 3:29 PM EDT.

    Service outageTLS certificate errors
    Trigger

    The origin certificate xpro-cert for xpro-web.odl.mit.edu, issued on 2026-03-05 with a 90-day validity period, expired at 3:29 PM EDT on 3 June 2026.

    Root cause
    Configuration change
    Component
    cert-managerCDN and edge networkAuthoritative DNS servers and records
    Contributing factor
    No configuration testsMonitoring blind spotChange with a delayed effect
    Detection

    Alerts for the xPRO production CMS and the main xPRO site fired at 3:38 PM EDT on the 503 errors, and a Rootly incident was created at 4:08 PM EDT.

    Remediation
    • Deleted and recreated the stale cert-manager certificate resources so that cert-manager issued a fresh origin certificate
    Action items
    • Remove the backend origin domain from the non-OpenEdX xPRO Fastly TLS subscription, so Fastly manages only xpro.mit.edu
    • Deploy the Pulumi update for the xPRO application in Production
    • Trigger cert-manager renewal of the xPRO certificate after the conflicting Fastly validation CNAME is removed
    • Consider tests or policy checks that keep cert-manager-managed origin domains out of Fastly TLS subscriptions
    • Consider alerting earlier on failed cert-manager CertificateRequests and Orders, so stale ACME failures are found before the certificate expires
    Lessons

    Reading the outage as a failed cert-manager ACME renewal was true but incomplete; the missing part was that Fastly had been set to manage validation for a backend origin domain that belonged to cert-manager. The team concluded that TLS certificates and DNS validation records need guardrails around ownership, and that public Fastly frontend domains and Kubernetes/APISIX origin domains must not be treated as interchangeable service domains in TLS automation.

    View case
  2. IL-0502 The JupyterHub at temple.2i2c.cloud stopped resolving in DNS after a wildcard record was added for a new cluster
    2i2c2025
    Symptoms

    On 20 August 2025 at 10:02 AM Pacific time, an automated uptime check on temple.2i2c.cloud started failing and paged 2i2c: the JupyterHub at temple.2i2c.cloud could not be reached, shortly before the community that uses it was to give a demo.

    DNS resolution failuresService outage
    Trigger

    At 9:21 AM, while setting up a new cluster for Temple, a wildcard DNS record *.temple.2i2c.cloud was added pointing to the new cluster's Ingress IP.

    Root cause
    Configuration change
    Component
    Authoritative DNS servers and records
    Contributing factor
    Unverified assumptionCache-masked failureUnreviewed defaults
    Detection

    Automated uptime alerts caught the failure at 10:02 AM, before the community reported it in a support ticket at 10:09 AM.

    Remediation
    • Added an explicit A record for temple.2i2c.cloud in Namecheap
    • Lowered the DNS TTL from the 30-minute default to 5 minutes
    • Offered the community member running the demo an /etc/hosts entry as a workaround
    Action items
    • Give everyone on the team access to Namecheap, or move to another DNS provider
    • Use a 5-minute TTL instead of 30 minutes for all new DNS entries
    • Document that adding a *.X.2i2c.cloud DNS entry requires adding an X.2i2c.cloud entry as well
    Lessons

    Adding a wildcard record for a subdomain stopped a broader wildcard from answering for that subdomain's own name, so a *.X.2i2c.cloud record needs an X.2i2c.cloud record alongside it. The 30-minute default TTL at Namecheap meant any DNS fix could take that long to propagate, and the engineer creating the original record had no access to the DNS provider.

    View case
  3. IL-0254 Square's internal DNS servers failed and Square products, services and payments were unavailable for about 15 hours
    Square2023
    Symptoms

    From 1:54 PM ET on 7 September 2023, Square products and services were unavailable, and at 2:02 PM ET engineers were notified of infrastructure issues and began investigating DNS. Square Online websites stayed up, but their customers could not make payments.

    DNS resolution failuresService outage
    Trigger

    At 11:04 AM ET a host-based firewall rule change was deployed to enable region communication, which increased the size of the firewall rules on nodes. Combined with a DNS service upgrade, it put unexpected load on the internal DNS servers. The timeline also lists a DNS zone change at 1:56 PM ET.

    Root cause
    Configuration change
    Component
    Authoritative DNS servers and recordsNode iptables rules
    Contributing factor
    No staged rolloutCache-masked failureCircular dependency
    Detection

    Engineers were notified of infrastructure issues at 2:02 PM ET, eight minutes after services became unavailable.

    Remediation
    • Shed network traffic to the DNS servers and started bringing up new DNS servers by hand
    • Deployed networking changes by hand to re-enable the authorization and access services
    • Deployed the networking changes by hand to all data centers
    • Rebuilt the DNS servers
    • Built a new configuration from the reverted firewall ruleset and pushed it to the DNS hosts
    Action items
    • Move the DNS infrastructure onto isolated infrastructure
    • Add monitoring and optimizations for critical networking infrastructure
    • Optimize dependencies between the deploy and platform infrastructure where feasible
    • Expand support for Offline Mode and improve communication about when it is available
    View case
Check your cluster for this

These 3 cases involve Authoritative DNS servers and records and CDN and edge network. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change3 cases

Component
Authoritative DNS servers and records3CDN and edge network1cert-manager1Node iptables rules1
Contributing factor
Cache-masked failure2Unverified assumption1Circular dependency1Unreviewed defaults1Change with a delayed effect1Monitoring blind spot1No configuration tests1No staged rollout1
Organisation
2i2c1MIT Open Learning1Square1
  1. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed
    MIT Open Learning2026
  2. IL-0502 The JupyterHub at temple.2i2c.cloud stopped resolving in DNS after a wildcard record was added for a new cluster
    2i2c2025
  3. IL-0254 Square's internal DNS servers failed and Square products, services and payments were unavailable for about 15 hours
    Square2023