Cases2 cases

1 to 2 of 2

1. IL-0502 The JupyterHub at temple.2i2c.cloud stopped resolving in DNS after a wildcard record was added for a new cluster

   2i2c2025

   Symptoms

   On 20 August 2025 at 10:02 AM Pacific time, an automated uptime check on temple.2i2c.cloud started failing and paged 2i2c: the JupyterHub at temple.2i2c.cloud could not be reached, shortly before the community that uses it was to give a demo.

   DNS resolution failuresService outage

   Trigger

   At 9:21 AM, while setting up a new cluster for Temple, a wildcard DNS record *.temple.2i2c.cloud was added pointing to the new cluster's Ingress IP.

   Root cause

   Configuration change

   Component

   Authoritative DNS servers and records

   Contributing factor

   Unverified assumptionCache-masked failureUnreviewed defaults

   Detection

   Automated uptime alerts caught the failure at 10:02 AM, before the community reported it in a support ticket at 10:09 AM.

   Remediation

   • Added an explicit A record for temple.2i2c.cloud in Namecheap
   • Lowered the DNS TTL from the 30-minute default to 5 minutes
   • Offered the community member running the demo an /etc/hosts entry as a workaround

   Action items

   • Give everyone on the team access to Namecheap, or move to another DNS provider
   • Use a 5-minute TTL instead of 30 minutes for all new DNS entries
   • Document that adding a *.X.2i2c.cloud DNS entry requires adding an X.2i2c.cloud entry as well

   Lessons

   Adding a wildcard record for a subdomain stopped a broader wildcard from answering for that subdomain's own name, so a *.X.2i2c.cloud record needs an X.2i2c.cloud record alongside it. The 30-minute default TTL at Namecheap meant any DNS fix could take that long to propagate, and the engineer creating the original record had no access to the DNS provider.

   View case
2. IL-0254 Square's internal DNS servers failed and Square products, services and payments were unavailable for about 15 hours

   Square2023

   Symptoms

   From 1:54 PM ET on 7 September 2023, Square products and services were unavailable, and at 2:02 PM ET engineers were notified of infrastructure issues and began investigating DNS. Square Online websites stayed up, but their customers could not make payments.

   DNS resolution failuresService outage

   Trigger

   At 11:04 AM ET a host-based firewall rule change was deployed to enable region communication, which increased the size of the firewall rules on nodes. Combined with a DNS service upgrade, it put unexpected load on the internal DNS servers. The timeline also lists a DNS zone change at 1:56 PM ET.

   Root cause

   Configuration change

   Component

   Authoritative DNS servers and recordsNode iptables rules

   Contributing factor

   No staged rolloutCache-masked failureCircular dependency

   Detection

   Engineers were notified of infrastructure issues at 2:02 PM ET, eight minutes after services became unavailable.

   Remediation

   • Shed network traffic to the DNS servers and started bringing up new DNS servers by hand
   • Deployed networking changes by hand to re-enable the authorization and access services
   • Deployed the networking changes by hand to all data centers
   • Rebuilt the DNS servers
   • Built a new configuration from the reverted firewall ruleset and pushed it to the DNS hosts

   Action items

   • Move the DNS infrastructure onto isolated infrastructure
   • Add monitoring and optimizations for critical networking infrastructure
   • Optimize dependencies between the deploy and platform infrastructure where feasible
   • Expand support for Offline Mode and improve communication about when it is available

   View case

By root cause1