Cases5 cases

1 to 5 of 5

1. IL-0502 The JupyterHub at temple.2i2c.cloud stopped resolving in DNS after a wildcard record was added for a new cluster

   2i2c2025

   Symptoms

   On 20 August 2025 at 10:02 AM Pacific time, an automated uptime check on temple.2i2c.cloud started failing and paged 2i2c: the JupyterHub at temple.2i2c.cloud could not be reached, shortly before the community that uses it was to give a demo.

   DNS resolution failuresService outage

   Trigger

   At 9:21 AM, while setting up a new cluster for Temple, a wildcard DNS record *.temple.2i2c.cloud was added pointing to the new cluster's Ingress IP.

   Root cause

   Configuration change

   Component

   Authoritative DNS servers and records

   Contributing factor

   Unverified assumptionCache-masked failureUnreviewed defaults

   Detection

   Automated uptime alerts caught the failure at 10:02 AM, before the community reported it in a support ticket at 10:09 AM.

   Remediation

   • Added an explicit A record for temple.2i2c.cloud in Namecheap
   • Lowered the DNS TTL from the 30-minute default to 5 minutes
   • Offered the community member running the demo an /etc/hosts entry as a workaround

   Action items

   • Give everyone on the team access to Namecheap, or move to another DNS provider
   • Use a 5-minute TTL instead of 30 minutes for all new DNS entries
   • Document that adding a *.X.2i2c.cloud DNS entry requires adding an X.2i2c.cloud entry as well

   Lessons

   Adding a wildcard record for a subdomain stopped a broader wildcard from answering for that subdomain's own name, so a *.X.2i2c.cloud record needs an X.2i2c.cloud record alongside it. The 30-minute default TTL at Namecheap meant any DNS fix could take that long to propagate, and the engineer creating the original record had no access to the DNS provider.

   View case
2. IL-0533 Names in chinacloudapp.cn and chinacloudsites.cn stopped resolving as DNS caches expired, breaking Azure China services

   Microsoft Azure2024

   Symptoms

   From 03:45 CST on 23 April 2024, FQDNs in two system domains of Azure in China, chinacloudapp.cn and chinacloudsites.cn, gradually stopped resolving, and customers in the Azure China regions had trouble connecting to many Azure services and managing resources. Internal monitors alerted engineers to the degradation at 04:37 CST.

   DNS resolution failuresHigh error rateMonitoring outage

   Trigger

   At 03:45 CST a configuration change made through a domain registrar, as a step of a domain decommissioning workflow, updated the nameservers of chinacloudapp.cn and chinacloudsites.cn to a set of inactive servers.

   Root cause

   Configuration change

   Component

   Authoritative DNS servers and records

   Contributing factor

   Unverified assumptionCache-masked failure

   Detection

   Internal monitors alerted engineers to the degradation in the Azure China Cloud at 04:37 CST and created an incident, which the engineering team acknowledged at 04:39 CST.

   Remediation

   • Reverted the two domains to their previous working nameservers

   Action items

   • Suspended further runs of the domain lifecycle process until the management tool covers Azure in China
   • Make the validation process of domain lifecycle management include signals from all cloud regions
   • Add validations that get nameserver information by resolving the zone directly over the Internet

   View case
3. IL-0549 Media, attachments, Git LFS and build artifacts failed for Atlassian in us-west-1 when an old network stack was removed

   Atlassian2023

   Symptoms

   From 02:36 UTC on 15 May 2023, Atlassian products with services hosted in the us-west-1 region could not store or retrieve data assets such as media, attachments and build artifacts. Automated monitoring detected it within three minutes, and the status page reported intermittent errors.

   High error rateStale endpoint routing

   Trigger

   An internal service was migrated to a new network segment as part of an initiative to harden the partitions between network segments, and the old network stack was then removed.

   Root cause

   Configuration change

   Component

   Authoritative DNS servers and records

   Contributing factor

   Unverified assumptionUnrepresentative staging

   Detection

   Automated monitoring detected the incident within three minutes.

   Remediation

   • Rolled back the network migration change, returning Atlassian systems to a known good state

   Action items

   • Review the systems that decommission service stacks and check that customer traffic is no longer served before a stack is decommissioned
   • Add steps to the service network migration process that identify associated DNS records needing attention

   Lessons

   Atlassian said the migration used blue/green deployment to limit this kind of risk, but adequate verification was not run before the old stack was decommissioned, and moving services between network segments is rare and hard to replicate in test.

   View case
4. IL-0250 slack.com stopped resolving on some DNSSEC-validating resolvers for about 24 hours after DNSSEC signing was turned on

   Slack2021

   Symptoms

   Soon after Slack published the DS record for slack.com at its DNS registrar on 30 September 2021, the Customer Experience team started getting user reports of "ERR_NAME_NOT_RESOLVED" errors in Slack clients. Fewer than 1% of online users were affected, and most reports came from users of one private corporate recursive resolver and of Google Public DNS (8.8.8.8).

   DNS resolution failures

   Trigger

   On 30 September 2021 the Traffic Engineering team enabled DNSSEC signing on the slack.com zone in Amazon Route 53 and on its sub-delegated zones in NS1, and after a three-hour soak published the slack.com DS record at the DNS registrar.

   Root cause

   Software defect

   Component

   Authoritative DNS servers and recordsRecursive DNS resolver serviceTerraform

   Contributing factor

   Unrepresentative stagingUnverified assumptionHarmful manual actionIneffective rollbackLimited debug access

   Detection

   Customer Experience received user reports of "ERR_NAME_NOT_RESOLVED" errors in Slack clients, and customers sent NetLog captures through support cases. After signing was rolled back, Slack's external DNS and DNSSEC resolution probes alerted the Traffic team.

   Remediation

   • Removed the slack.com DS record from the DNS registrar
   • Rolled back DNSSEC signing on the slack.com authoritative and delegated zones, which made resolution worse
   • Asked major ISPs and public DNS resolver operators to flush their cached slack.com records
   • Flushed the slack.com records on Google's 8.8.8.8 and Cloudflare's 1.1.1.1 through their public flush sites

   Lessons

   The DNSSEC rollout had been tested only on domains without a wildcard record, so the Route 53 wildcard NSEC bug first appeared on slack.com; Slack names not testing a wildcard domain first as an oversight. Removing the DS record at the registrar does not stop validation at once, because resolvers cache the DS record from the .com zone for up to 24 hours, and removing the DNSKEY while that record is still cached makes every validating resolver fail. Managing DNS through Terraform hid the console warning about disabling DNSSEC. Re-enabling signing would neither have helped nor hurt, because Route 53 generates a new ZSK each time signing is enabled.

   View case
5. IL-0351 Heroku apps in the EU and US had slow and failed requests as DNS round-robin sent traffic to single router hosts

   Heroku2017

   Symptoms

   Between 3 and 7 June 2017, Heroku customers in both the EU and US regions had periods of high latency: some requests failed and others took an excessive time to pass through the Heroku Routing Layer. All customers in the EU and US were affected at various times.

   High latencyHigh error rateLoad imbalance

   Root cause

   DNS or network fault

   Component

   Authoritative DNS servers and recordsCloud load balancer

   Contributing factor

   Unverified assumption

   Remediation

   • Found the sources of high-volume traffic and worked with those apps' owners to distribute their requests better
   • Replaced the legacy DNS records with new, corrected DNS records

   Action items

   • Make monitoring alert engineers sooner to unevenly distributed traffic
   • Investigate how to configure DNS and load balancers to reduce the effect of high-volume and bursting traffic
   • Work with caching Add-on partners on the DNS configuration issues that affect them, to achieve proper load balancing

   View case

By root cause3