Incident Library
Applied filters SymptomDNS resolution failuresFactorNo safeguard on destructive changeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0260 zoom.us stopped resolving for about two hours after GoDaddy Registry placed a server block on the domain
    Zoom2025
    Symptoms

    On 16 April 2025 customers in the US and International regions could not reach Zoom services because lookups of the zoom.us domain failed. Starting, joining or scheduling a meeting needs a DNS lookup, so none of those actions completed; people already in a Zoom meeting or on a Zoom Phone call were not affected.

    DNS resolution failuresService outage
    Trigger

    A communication error between Zoom's domain registrar, Markmonitor, and GoDaddy Registry led GoDaddy Registry to place a server block on zoom.us by mistake, which shut the domain down.

    Root cause
    Dependency failure
    Component
    Authoritative DNS servers and records
    Contributing factor
    No safeguard on destructive change
    Remediation
    • Identified and removed the server block on zoom.us together with Markmonitor and GoDaddy
    • Asked users who still had connection problems to flush their DNS cache and reconnect
    Action items
    • Put a registry lock on zoom.us that restricts server block commands on the domain
    • Implement additional technical changes and processes with GoDaddy Registry and Markmonitor so the issue does not recur
    View case
  2. IL-0363 static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch crates
    Rust Project2023
    Symptoms

    On 25 January 2023, shortly after a deployment to the crates.io production infrastructure at 09:15 UTC, the DNS record for static.crates.io failed to resolve for an estimated 10-15 minutes. Builds failed for users because crates could not be downloaded.

    DNS resolution failures
    Trigger

    A Terraform deployment to production contained a new TLS certificate for the current CDN of static.crates.io and updated DNS records; Terraform removed the current certificate and DNS records first and recreated the records only after the new certificate had been issued.

    Root cause
    Configuration change
    Component
    TerraformAuthoritative DNS servers and recordsCDN and edge network
    Contributing factor
    No safeguard on destructive changeUnrepresentative staging
    Remediation
    • Terraform finished the deployment and created a new DNS record for static.crates.io, which ended the outage
    Action items
    • Document the process of deploying changes to production, including how to pick the Git commit and how to review the changeset
    • Deploy changes developed and tested in isolation on staging individually and in sequence to production, and add this to the documentation
    • Deploy changes to production when they are merged into the main branch, so the configuration in Git and what is deployed do not drift
    Lessons

    Deploying the TLS certificate change and the DNS record change separately would have avoided the outage. Changes tested one at a time on staging need to reach production one at a time and in order, the production deploy process needs to be written down so it can be improved, and merged changes need to be deployed promptly so Git and production do not drift apart.

    View case
  3. IL-0389 DNS queries from EC2 instances in the Seoul region failed for 84 minutes as the EC2 DNS resolver fleet lost hosts
    Amazon Web Services2018
    Symptoms

    On 22 November 2018 between 8:19 AM and 9:43 AM KST, DNS queries from within EC2 instances in the Seoul region (AP-NORTHEAST-2) failed, because the EC2 DNS resolver fleet that provides recursive DNS to EC2 instances had too few healthy hosts in service. Network connectivity of EC2 instances and DNS resolution outside EC2 instances were not affected.

    DNS resolution failures
    Trigger

    A configuration update incorrectly removed the setting that specifies the minimum number of healthy hosts for the EC2 DNS resolver fleet in AP-NORTHEAST-2.

    Root cause
    Configuration change
    Component
    VPC DNS resolver
    Contributing factor
    No configuration testsNo safeguard on destructive change
    Detection

    The engineering team was alerted to the DNS resolution issue at 8:21 AM KST.

    Remediation
    • Stopped more healthy hosts from being removed from the EC2 DNS resolver fleet
    • Restored the resolver fleet's capacity to its earlier level
    Action items
    • Check that every AWS region has the correct capacity settings for the EC2 DNS resolver service
    • Add semantic configuration validation to every EC2 DNS resolver configuration update
    • Throttle how much healthy host capacity can be removed from service each hour
    View case
Check your cluster for this

These 3 cases involve Authoritative DNS servers and records and CDN and edge network. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Dependency failure1 case

Component
Authoritative DNS servers and records
Contributing factor
No safeguard on destructive change
Organisation
Zoom
  1. IL-0260 zoom.us stopped resolving for about two hours after GoDaddy Registry placed a server block on the domain
    Zoom2025

Configuration change2 cases

Component
CDN and edge network1Authoritative DNS servers and records1Terraform1VPC DNS resolver1
Contributing factor
No safeguard on destructive change2No configuration tests1Unrepresentative staging1
Organisation
Amazon Web Services1Rust Project1
  1. IL-0363 static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch crates
    Rust Project2023
  2. IL-0389 DNS queries from EC2 instances in the Seoul region failed for 84 minutes as the EC2 DNS resolver fleet lost hosts
    Amazon Web Services2018