Cases2 cases

1 to 2 of 2

1. IL-0363 static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch crates

   Rust Project2023

   Symptoms

   On 25 January 2023, shortly after a deployment to the crates.io production infrastructure at 09:15 UTC, the DNS record for static.crates.io failed to resolve for an estimated 10-15 minutes. Builds failed for users because crates could not be downloaded.

   DNS resolution failures

   Trigger

   A Terraform deployment to production contained a new TLS certificate for the current CDN of static.crates.io and updated DNS records; Terraform removed the current certificate and DNS records first and recreated the records only after the new certificate had been issued.

   Root cause

   Configuration change

   Component

   TerraformAuthoritative DNS servers and recordsCDN and edge network

   Contributing factor

   No safeguard on destructive changeUnrepresentative staging

   Remediation

   • Terraform finished the deployment and created a new DNS record for static.crates.io, which ended the outage

   Action items

   • Document the process of deploying changes to production, including how to pick the Git commit and how to review the changeset
   • Deploy changes developed and tested in isolation on staging individually and in sequence to production, and add this to the documentation
   • Deploy changes to production when they are merged into the main branch, so the configuration in Git and what is deployed do not drift

   Lessons

   Deploying the TLS certificate change and the DNS record change separately would have avoided the outage. Changes tested one at a time on staging need to reach production one at a time and in order, the production deploy process needs to be written down so it can be improved, and merged changes need to be deployed promptly so Git and production do not drift apart.

   View case
2. IL-0250 slack.com stopped resolving on some DNSSEC-validating resolvers for about 24 hours after DNSSEC signing was turned on

   Slack2021

   Symptoms

   Soon after Slack published the DS record for slack.com at its DNS registrar on 30 September 2021, the Customer Experience team started getting user reports of "ERR_NAME_NOT_RESOLVED" errors in Slack clients. Fewer than 1% of online users were affected, and most reports came from users of one private corporate recursive resolver and of Google Public DNS (8.8.8.8).

   DNS resolution failures

   Trigger

   On 30 September 2021 the Traffic Engineering team enabled DNSSEC signing on the slack.com zone in Amazon Route 53 and on its sub-delegated zones in NS1, and after a three-hour soak published the slack.com DS record at the DNS registrar.

   Root cause

   Software defect

   Component

   Authoritative DNS servers and recordsRecursive DNS resolver serviceTerraform

   Contributing factor

   Unrepresentative stagingUnverified assumptionHarmful manual actionIneffective rollbackLimited debug access

   Detection

   Customer Experience received user reports of "ERR_NAME_NOT_RESOLVED" errors in Slack clients, and customers sent NetLog captures through support cases. After signing was rolled back, Slack's external DNS and DNSSEC resolution probes alerted the Traffic team.

   Remediation

   • Removed the slack.com DS record from the DNS registrar
   • Rolled back DNSSEC signing on the slack.com authoritative and delegated zones, which made resolution worse
   • Asked major ISPs and public DNS resolver operators to flush their cached slack.com records
   • Flushed the slack.com records on Google's 8.8.8.8 and Cloudflare's 1.1.1.1 through their public flush sites

   Lessons

   The DNSSEC rollout had been tested only on domains without a wildcard record, so the Route 53 wildcard NSEC bug first appeared on slack.com; Slack names not testing a wildcard domain first as an oversight. Removing the DS record at the registrar does not stop validation at once, because resolvers cache the DS record from the .com zone for up to 24 hours, and removing the DNSKEY while that record is still cached makes every validating resolver fail. Managing DNS through Terraform hid the console warning about disabling DNSSEC. Re-enabling signing would neither have helped nor hurt, because Route 53 generates a new ZSK each time signing is enabled.

   View case

By root cause2