Incident Library
Applied filters ComponentAuthoritative DNS servers and recordsFactorNo safeguard on destructive changeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0260 zoom.us stopped resolving for about two hours after GoDaddy Registry placed a server block on the domain
    Zoom2025
    Symptoms

    On 16 April 2025 customers in the US and International regions could not reach Zoom services because lookups of the zoom.us domain failed. Starting, joining or scheduling a meeting needs a DNS lookup, so none of those actions completed; people already in a Zoom meeting or on a Zoom Phone call were not affected.

    DNS resolution failuresService outage
    Trigger

    A communication error between Zoom's domain registrar, Markmonitor, and GoDaddy Registry led GoDaddy Registry to place a server block on zoom.us by mistake, which shut the domain down.

    Root cause
    Dependency failure
    Component
    Authoritative DNS servers and records
    Contributing factor
    No safeguard on destructive change
    Remediation
    • Identified and removed the server block on zoom.us together with Markmonitor and GoDaddy
    • Asked users who still had connection problems to flush their DNS cache and reconnect
    Action items
    • Put a registry lock on zoom.us that restricts server block commands on the domain
    • Implement additional technical changes and processes with GoDaddy Registry and Markmonitor so the issue does not recur
    View case
  2. IL-0363 static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch crates
    Rust Project2023
    Symptoms

    On 25 January 2023, shortly after a deployment to the crates.io production infrastructure at 09:15 UTC, the DNS record for static.crates.io failed to resolve for an estimated 10-15 minutes. Builds failed for users because crates could not be downloaded.

    DNS resolution failures
    Trigger

    A Terraform deployment to production contained a new TLS certificate for the current CDN of static.crates.io and updated DNS records; Terraform removed the current certificate and DNS records first and recreated the records only after the new certificate had been issued.

    Root cause
    Configuration change
    Component
    TerraformAuthoritative DNS servers and recordsCDN and edge network
    Contributing factor
    No safeguard on destructive changeUnrepresentative staging
    Remediation
    • Terraform finished the deployment and created a new DNS record for static.crates.io, which ended the outage
    Action items
    • Document the process of deploying changes to production, including how to pick the Git commit and how to review the changeset
    • Deploy changes developed and tested in isolation on staging individually and in sequence to production, and add this to the documentation
    • Deploy changes to production when they are merged into the main branch, so the configuration in Git and what is deployed do not drift
    Lessons

    Deploying the TLS certificate change and the DNS record change separately would have avoided the outage. Changes tested one at a time on staging need to reach production one at a time and in order, the production deploy process needs to be written down so it can be improved, and merged changes need to be deployed promptly so Git and production do not drift apart.

    View case
Check your cluster for this

These 2 cases involve Authoritative DNS servers and records and CDN and edge network. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Dependency failure1 case

Component
Authoritative DNS servers and records
Contributing factor
No safeguard on destructive change
Organisation
Zoom
  1. IL-0260 zoom.us stopped resolving for about two hours after GoDaddy Registry placed a server block on the domain
    Zoom2025

Configuration change1 case

Component
CDN and edge networkAuthoritative DNS servers and recordsTerraform
Contributing factor
No safeguard on destructive changeUnrepresentative staging
Organisation
Rust Project
  1. IL-0363 static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch crates
    Rust Project2023