Cases3 cases

1 to 3 of 3

1. IL-0260 zoom.us stopped resolving for about two hours after GoDaddy Registry placed a server block on the domain

   Zoom2025

   Symptoms

   On 16 April 2025 customers in the US and International regions could not reach Zoom services because lookups of the zoom.us domain failed. Starting, joining or scheduling a meeting needs a DNS lookup, so none of those actions completed; people already in a Zoom meeting or on a Zoom Phone call were not affected.

   DNS resolution failuresService outage

   Trigger

   A communication error between Zoom's domain registrar, Markmonitor, and GoDaddy Registry led GoDaddy Registry to place a server block on zoom.us by mistake, which shut the domain down.

   Root cause

   Dependency failure

   Component

   Authoritative DNS servers and records

   Contributing factor

   No safeguard on destructive change

   Remediation

   • Identified and removed the server block on zoom.us together with Markmonitor and GoDaddy
   • Asked users who still had connection problems to flush their DNS cache and reconnect

   Action items

   • Put a registry lock on zoom.us that restricts server block commands on the domain
   • Implement additional technical changes and processes with GoDaddy Registry and Markmonitor so the issue does not recur

   View case
2. IL-0363 static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch crates

   Rust Project2023

   Symptoms

   On 25 January 2023, shortly after a deployment to the crates.io production infrastructure at 09:15 UTC, the DNS record for static.crates.io failed to resolve for an estimated 10-15 minutes. Builds failed for users because crates could not be downloaded.

   DNS resolution failures

   Trigger

   A Terraform deployment to production contained a new TLS certificate for the current CDN of static.crates.io and updated DNS records; Terraform removed the current certificate and DNS records first and recreated the records only after the new certificate had been issued.

   Root cause

   Configuration change

   Component

   TerraformAuthoritative DNS servers and recordsCDN and edge network

   Contributing factor

   No safeguard on destructive changeUnrepresentative staging

   Remediation

   • Terraform finished the deployment and created a new DNS record for static.crates.io, which ended the outage

   Action items

   • Document the process of deploying changes to production, including how to pick the Git commit and how to review the changeset
   • Deploy changes developed and tested in isolation on staging individually and in sequence to production, and add this to the documentation
   • Deploy changes to production when they are merged into the main branch, so the configuration in Git and what is deployed do not drift

   Lessons

   Deploying the TLS certificate change and the DNS record change separately would have avoided the outage. Changes tested one at a time on staging need to reach production one at a time and in order, the production deploy process needs to be written down so it can be improved, and merged changes need to be deployed promptly so Git and production do not drift apart.

   View case
3. IL-0389 DNS queries from EC2 instances in the Seoul region failed for 84 minutes as the EC2 DNS resolver fleet lost hosts

   Amazon Web Services2018

   Symptoms

   On 22 November 2018 between 8:19 AM and 9:43 AM KST, DNS queries from within EC2 instances in the Seoul region (AP-NORTHEAST-2) failed, because the EC2 DNS resolver fleet that provides recursive DNS to EC2 instances had too few healthy hosts in service. Network connectivity of EC2 instances and DNS resolution outside EC2 instances were not affected.

   DNS resolution failures

   Trigger

   A configuration update incorrectly removed the setting that specifies the minimum number of healthy hosts for the EC2 DNS resolver fleet in AP-NORTHEAST-2.

   Root cause

   Configuration change

   Component

   VPC DNS resolver

   Contributing factor

   No configuration testsNo safeguard on destructive change

   Detection

   The engineering team was alerted to the DNS resolution issue at 8:21 AM KST.

   Remediation

   • Stopped more healthy hosts from being removed from the EC2 DNS resolver fleet
   • Restored the resolver fleet's capacity to its earlier level

   Action items

   • Check that every AWS region has the correct capacity settings for the EC2 DNS resolver service
   • Add semantic configuration validation to every EC2 DNS resolver configuration update
   • Throttle how much healthy host capacity can be removed from service each hour

   View case

By root cause2