Cases1 case

1 to 1 of 1

1. IL-0340 Cloudflare's 1.1.1.1 resolver returned SERVFAIL to some valid DNS queries for four hours

   Cloudflare2023

   Symptoms

   From 07:00 to 11:00 UTC on 4 October 2023, some users of Cloudflare's 1.1.1.1 DNS resolver received SERVFAIL responses to valid queries. This reached users of WARP and Zero Trust and of third-party DNS resolvers that use 1.1.1.1, and the first external reports of unexpected SERVFAILs arrived at 7:57 UTC.

   DNS resolution failuresHigh error rate

   Trigger

   On 21 September 2023, as a planned change in root zone management, a ZONEMD resource record, in effect a checksum of the zone's contents, was added to the root zone for the first time. On 4 October at 07:00 UTC the DNSSEC signatures in the last root zone copy that 1.1.1.1 had loaded expired.

   Root cause

   Software defect

   Component

   Recursive DNS resolver service

   Contributing factor

   Cache-masked failureUnsafe failure behaviourMonitoring blind spotGap in automated tests

   Detection

   External reports of unexpected SERVFAILs started coming in at 7:57 UTC, and Cloudflare declared an internal incident at 8:03 UTC. Nothing had alerted on static_zone serving a stale root zone during the two weeks before.

   Remediation

   • Deployed an override rule meant to stop 1.1.1.1 from answering with the stale root zone, which did not cover forwarded queries
   • Stopped 1.1.1.1 from preloading the root zone file entirely

   Action items

   • Add alerts for when static_zone serves a stale root zone file
   • Re-evaluate how the root zone is ingested and distributed internally so new record types are handled
   • Improve test coverage for what happens when the root zone fails to parse
   • Limit how long stale copies of the root zone can be used, following RFC 8806

   Lessons

   Cloudflare's main lesson is never to assume an input will not change. Modern systems pull in long chains of libraries that may not be ready when their input changes, so they need tests that catch regressions and components that fail gracefully on new input, and format changes in DNS and BGP should be expected to have an impact. Cloudflare also writes that with better monitoring of the stale root zone, the existing caching would have prevented any impact.

   View case

By root cause1