Incident Library

IL-0340 Cloudflare · 2023-10-04networking

Cloudflare's 1.1.1.1 resolver returned SERVFAIL to some valid DNS queries for four hours

Cloudflare's recursive DNS infrastructure keeps its own copy of the DNS root zone, which is signed with DNSSEC, so that it can resolve names faster and still work if the root servers cannot be reached. Software in Cloudflare's core network retrieves new root zone versions, normally published twice a day, and redistributes them to data centers worldwide, where a WebAssembly app called static_zone running on top of the 1.1.1.1 DNS logic loads and serves them. static_zone keeps the latest version in memory and drops the old one only after a new one parses. Because Cloudflare distributed the root zone internally in its text presentation format rather than the binary format, the parser library at the edge returned a parser error on the unknown ZONEMD record instead of skipping it.

After the ZONEMD record appeared on 21 September, the root zone was still retrieved and distributed as usual, but no 1.1.1.1 server could load the new versions. Servers that had a version in memory kept using the one from before the change indefinitely; servers started after the change had nothing to fall back on and sent each query to the root servers directly. At 07:00 UTC on 4 October the DNSSEC signatures in the 21 September copy expired, and servers still using it could no longer validate DNSSEC and answered SERVFAIL. Cloudflare declared an internal incident at 8:03 UTC. An override rule at 8:50 UTC was meant to stop answers from the stale root zone, but its tag was not forwarded with queries passed between nodes. At 10:30 UTC the team stopped 1.1.1.1 from preloading the root zone entirely, responses were normal again at 10:32 UTC, and the incident was closed at 11:02 UTC.

Severity
–
Scope
DNS resolution through 1.1.1.1 and products built on it, including WARP, Zero Trust and third-party DNS resolvers that use 1.1.1.1; SERVFAIL peaked at 15% of all queries
Users affected
some users of 1.1.1.1
Region
mainly larger data centers such as Ashburn, Virginia; Frankfurt, Germany; and Singapore
Time to detect
1h 3m
Time to resolve
3h 32m

Symptoms

From 07:00 to 11:00 UTC on 4 October 2023, some users of Cloudflare's 1.1.1.1 DNS resolver received SERVFAIL responses to valid queries. This reached users of WARP and Zero Trust and of third-party DNS resolvers that use 1.1.1.1, and the first external reports of unexpected SERVFAILs arrived at 7:57 UTC.

Normally about 3% of 1.1.1.1 answers are SERVFAIL, from real DNSSEC problems, unreachable or slow authoritative servers and similar causes. During the incident the rate of SERVFAIL responses grew by 12% and peaked at 15% of all queries. The failures were not spread evenly: they were concentrated in the larger data centers such as Ashburn, Virginia; Frankfurt, Germany; and Singapore, because many servers there had not restarted the 1.1.1.1 service in the previous two weeks. Serve stale (RFC 8767), which keeps answering popular records from cache after their TTL, prevented a total outage.

The failure surfaced two weeks after its cause. On 21 September the root zone started to carry a new record type, ZONEMD, and the resolver software could not parse it, so servers that already held a copy of the root zone in memory kept using it; that copy had last been pulled at 6:30 UTC on 21 September. Nothing went wrong until the DNSSEC signatures in that copy expired at 07:00 UTC on 4 October, after which DNSSEC validation failed. Servers restarted in between had no copy to fall back on and resolved through the root servers directly, so they did not fail.

The first fix did not work. At 8:50 UTC engineers deployed an override rule, phase = pre-cache set-tag rec_disable_static, to make the static_zone app stop answering from the stale root zone. The tag rec_disable_static was not carried on queries that a node forwards to another node when it misses its own cache, so static_zone kept serving stale data until the team stopped 1.1.1.1 from preloading the root zone at all at 10:30 UTC.

Symptom DNS resolution failuresHigh error rate

Root cause

Root cause Software defect

Component Recursive DNS resolver service
Contributing factor Cache-masked failureUnsafe failure behaviourMonitoring blind spotGap in automated tests
Trigger
On 21 September 2023, as a planned change in root zone management, a ZONEMD resource record, in effect a checksum of the zone's contents, was added to the root zone for the first time. On 4 October at 07:00 UTC the DNSSEC signatures in the last root zone copy that 1.1.1.1 had loaded expired.

Detection

External reports of unexpected SERVFAILs started coming in at 7:57 UTC, and Cloudflare declared an internal incident at 8:03 UTC. Nothing had alerted on static_zone serving a stale root zone during the two weeks before.

Remediation

Action items

Lessons

Cloudflare's main lesson is never to assume an input will not change. Modern systems pull in long chains of libraries that may not be ready when their input changes, so they need tests that catch regressions and components that fail gracefully on new input, and format changes in DNS and BGP should be expected to have an impact. Cloudflare also writes that with better monitoring of the stale root zone, the existing caching would have prevented any impact.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Recursive DNS resolver service among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.