Incident Library

IL-0250 Slack · 2021-09-30networkingaws

slack.com stopped resolving on some DNSSEC-validating resolvers for about 24 hours after DNSSEC signing was turned on

Slack uses Amazon Route 53 as the authoritative DNS server for its public domains and delegates some subzones to NS1. DNSSEC, which FedRAMP Moderate requires, can only be enabled for a whole domain, not per subdomain or subzone. Every other critical Slack domain had run with DNSSEC for at least a month, and 30 September 2021 was the third attempt on slack.com, after attempts on 7 and 8 September had been rolled back. The team validated each step with external DNS and DNSSEC monitoring from many resolvers, dnsviz.net and Verisign DNSSEC Debugger tests, dnschecker.org tests and dig, and all checks passed before and after the DS record was published.

The root cause was a bug in how Route 53 answered for wildcard records. slack.com has a *.slack.com wildcard record, which none of the domains rolled out before had. For a wildcard name and a record type that does not exist, Route 53 returned a signed NSEC record that did not list the types that do exist. When a client asked for the AAAA record of app.slack.com, which Slack does not publish, some resolvers, including Google Public DNS, cached that NSEC answer and concluded that no A record existed either, so they returned NODATA for the A lookup. The cached answer poisoned only one regional cache at a time, which is why only some users of an affected resolver saw failures. Slack's first attempts to reproduce the failure with a copy of the slack.com setup failed; after the NetLogs pointed to the wildcard and the Route 53 team linked it to the NSEC bug, a DNSSEC-enabled test zone reproduced it against Google Public DNS but not against Cloudflare Public DNS (1.1.1.1). Slack concluded that reusing a cached NSEC record this way is protocol compliant, as in the optional Aggressive NSEC (RFC 8198) that Google implements. Route 53 later rolled out a fix.

During the incident the team removed the DS record from the registrar and then, when nothing improved, removed DNSSEC signing from the zones through Terraform, which also meant the team did not see the Route 53 console warning about disabling DNSSEC. Based on expert advice the team believed that DS records in the .com zone were not cached, but the .com zone asks resolvers to cache the slack.com DS record for 24 hours, a value the zone owner cannot change (Google's 8.8.8.8 caps it at 6 hours). Resolvers that still held the DS record could no longer validate the unsigned zone and returned SERVFAIL. Slack asked major ISPs and public resolver operators to flush their slack.com records and used the public flush pages of 8.8.8.8 and 1.1.1.1. Re-signing the zone was considered but not done; Route 53 later confirmed it would have generated a new zone signing key (ZSK), so validation would have kept failing. Client telemetry sent to slackb.com, a separate domain, kept working and showed the impact. Error rates returned to normal as caches were flushed and the 24-hour DS TTL expired.

Severity
–
Scope
Resolution of slack.com names such as app.slack.com on some recursive resolvers, so affected Slack clients could not connect for about 24 hours
Users affected
Less than 1% of Slack's online users
Region
–
Time to detect
–
Time to resolve
24h

Symptoms

Soon after Slack published the DS record for slack.com at its DNS registrar on 30 September 2021, the Customer Experience team started getting user reports of "ERR_NAME_NOT_RESOLVED" errors in Slack clients. Fewer than 1% of online users were affected, and most reports came from users of one private corporate recursive resolver and of Google Public DNS (8.8.8.8).

The error came from NODATA DNS responses returned by some recursive resolvers. NetLog captures that customers exported from the Chromium-based Slack clients showed slack.com resolving successfully and app.slack.com failing with ERR_NAME_NOT_RESOLVED a moment later. Slack's own DNSSEC probes had gone green after the DS record was published, and the impact looked far smaller than a real Google DNS problem would cause.

The team first removed the DS record from the registrar, but after nearly an hour the error rate had neither risen nor fallen. The team then rolled back DNSSEC signing on the slack.com zones, and the situation became much worse: the Traffic team was paged for slack.com resolution failures from many resolvers around the world, for example 9.9.9.9, and every DNSSEC-validating resolver that had recently cached the slack.com DS record returned SERVFAIL for all slack.com lookups.

Resolution recovered slowly as resolver operators flushed their caches. Some hours in, slack.com resolved correctly for the vast majority of customers, but users of smaller resolvers that had not flushed their slack.com records kept failing until the 24-hour TTL on the DS record in the .com zone ran out.

Symptom DNS resolution failures

Root cause

Root cause Software defect

Component Authoritative DNS servers and recordsRecursive DNS resolver serviceTerraform
Contributing factor Unrepresentative stagingUnverified assumptionHarmful manual actionIneffective rollbackLimited debug access
Trigger
On 30 September 2021 the Traffic Engineering team enabled DNSSEC signing on the slack.com zone in Amazon Route 53 and on its sub-delegated zones in NS1, and after a three-hour soak published the slack.com DS record at the DNS registrar.

Detection

Customer Experience received user reports of "ERR_NAME_NOT_RESOLVED" errors in Slack clients, and customers sent NetLog captures through support cases. After signing was rolled back, Slack's external DNS and DNSSEC resolution probes alerted the Traffic team.

Remediation

Action items

–

Lessons

The DNSSEC rollout had been tested only on domains without a wildcard record, so the Route 53 wildcard NSEC bug first appeared on slack.com; Slack names not testing a wildcard domain first as an oversight. Removing the DS record at the registrar does not stop validation at once, because resolvers cache the DS record from the .com zone for up to 24 hours, and removing the DNSKEY while that record is still cached makes every validating resolver fail. Managing DNS through Terraform hid the console warning about disabling DNSSEC. Re-enabling signing would neither have helped nor hurt, because Route 53 generates a new ZSK each time signing is enabled.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Authoritative DNS servers and records, Recursive DNS resolver service, Terraform among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.