Incident Library
Applied filters Root causeSoftware defectComponentTerraformFactorHarmful manual actionClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0250 slack.com stopped resolving on some DNSSEC-validating resolvers for about 24 hours after DNSSEC signing was turned on
    Slack2021
    Symptoms

    Soon after Slack published the DS record for slack.com at its DNS registrar on 30 September 2021, the Customer Experience team started getting user reports of "ERR_NAME_NOT_RESOLVED" errors in Slack clients. Fewer than 1% of online users were affected, and most reports came from users of one private corporate recursive resolver and of Google Public DNS (8.8.8.8).

    DNS resolution failures
    Trigger

    On 30 September 2021 the Traffic Engineering team enabled DNSSEC signing on the slack.com zone in Amazon Route 53 and on its sub-delegated zones in NS1, and after a three-hour soak published the slack.com DS record at the DNS registrar.

    Root cause
    Software defect
    Component
    Authoritative DNS servers and recordsRecursive DNS resolver serviceTerraform
    Contributing factor
    Unrepresentative stagingUnverified assumptionHarmful manual actionIneffective rollbackLimited debug access
    Detection

    Customer Experience received user reports of "ERR_NAME_NOT_RESOLVED" errors in Slack clients, and customers sent NetLog captures through support cases. After signing was rolled back, Slack's external DNS and DNSSEC resolution probes alerted the Traffic team.

    Remediation
    • Removed the slack.com DS record from the DNS registrar
    • Rolled back DNSSEC signing on the slack.com authoritative and delegated zones, which made resolution worse
    • Asked major ISPs and public DNS resolver operators to flush their cached slack.com records
    • Flushed the slack.com records on Google's 8.8.8.8 and Cloudflare's 1.1.1.1 through their public flush sites
    Lessons

    The DNSSEC rollout had been tested only on domains without a wildcard record, so the Route 53 wildcard NSEC bug first appeared on slack.com; Slack names not testing a wildcard domain first as an oversight. Removing the DS record at the registrar does not stop validation at once, because resolvers cache the DS record from the .com zone for up to 24 hours, and removing the DNSKEY while that record is still cached makes every validating resolver fail. Managing DNS through Terraform hid the console warning about disabling DNSSEC. Re-enabling signing would neither have helped nor hurt, because Route 53 generates a new ZSK each time signing is enabled.

    View case
Check your cluster for this

This case involves Authoritative DNS servers and records and Recursive DNS resolver service. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
Authoritative DNS servers and recordsRecursive DNS resolver serviceTerraform
Contributing factor
Unverified assumptionLimited debug accessHarmful manual actionIneffective rollbackUnrepresentative staging
Organisation
Slack
  1. IL-0250 slack.com stopped resolving on some DNSSEC-validating resolvers for about 24 hours after DNSSEC signing was turned on
    Slack2021