The JupyterHub at temple.2i2c.cloud stopped resolving in DNS after a wildcard record was added for a new cluster

Symptoms

On 20 August 2025 at 10:02 AM Pacific time, an automated uptime check on temple.2i2c.cloud started failing and paged 2i2c: the JupyterHub at temple.2i2c.cloud could not be reached, shortly before the community that uses it was to give a demo.

The alert, INCIDENT #1194, read "An uptime check on two-eye-two-see Uptime Check URL labels {project_id=two-eye-two-see, host=temple.2i2c.cloud} is failing." Running `dig temple.2i2c.cloud` returned NOERROR but no entries, so the hub's name was not resolving. A community member opened a support ticket at 10:09 AM.

The DNS record that caused the failure had been added at 9:21 AM, about 40 minutes earlier, without any visible effect at the time; the team put the delay down to DNS caching. Because the default TTL on Namecheap records is 30 minutes, any fix could take that long to propagate. DNS resolved properly for the engineer working on the issue only at 10:47 AM.

Symptom DNS resolution failuresService outage

Root cause

Trigger

At 9:21 AM, while setting up a new cluster for Temple, a wildcard DNS record *.temple.2i2c.cloud was added pointing to the new cluster's Ingress IP.

Detection

Automated uptime alerts caught the failure at 10:02 AM, before the community reported it in a support ticket at 10:09 AM.

Remediation

• Added an explicit A record for temple.2i2c.cloud in Namecheap
• Lowered the DNS TTL from the 30-minute default to 5 minutes
• Offered the community member running the demo an /etc/hosts entry as a workaround

Action items

• Give everyone on the team access to Namecheap, or move to another DNS provider
• Use a 5-minute TTL instead of 30 minutes for all new DNS entries
• Document that adding a *.X.2i2c.cloud DNS entry requires adding an X.2i2c.cloud entry as well

Lessons

Adding a wildcard record for a subdomain stopped a broader wildcard from answering for that subdomain's own name, so a *.X.2i2c.cloud record needs an X.2i2c.cloud record alongside it. The 30-minute default TTL at Namecheap meant any DNS fix could take that long to propagate, and the engineer creating the original record had no access to the DNS provider.

Sources

• https://github.com/2i2c-org/incident-reports/blob/main/reports/2025-08-20-temple.2i2c.cloud-unreachable.pdf · github

References

• https://github.com/2i2c-org/infrastructure/issues/6487
• https://github.com/2i2c-org/infrastructure/issues/6646

Related cases

• IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failedMIT Open Learning · 2026 · sev1

  same root cause · shared Authoritative DNS servers and records

• IL-0533 Names in chinacloudapp.cn and chinacloudsites.cn stopped resolving as DNS caches expired, breaking Azure China servicesMicrosoft Azure · 2024

  same root cause · shared Authoritative DNS servers and records

• IL-0254 Square's internal DNS servers failed and Square products, services and payments were unavailable for about 15 hoursSquare · 2023

  same root cause · shared Authoritative DNS servers and records

• IL-0549 Media, attachments, Git LFS and build artifacts failed for Atlassian in us-west-1 when an old network stack was removedAtlassian · 2023

  same root cause · shared Authoritative DNS servers and records

• IL-0363 static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch cratesRust Project · 2023

  same root cause · shared Authoritative DNS servers and records

Check your cluster for this

For AI tools and scripts

• This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0502
• The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0502/similar
• MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.