Incident Library
Applied filters Root causeSoftware defectOrganisationCloudflareClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0340 Cloudflare's 1.1.1.1 resolver returned SERVFAIL to some valid DNS queries for four hours
    Cloudflare2023
    Symptoms

    From 07:00 to 11:00 UTC on 4 October 2023, some users of Cloudflare's 1.1.1.1 DNS resolver received SERVFAIL responses to valid queries. This reached users of WARP and Zero Trust and of third-party DNS resolvers that use 1.1.1.1, and the first external reports of unexpected SERVFAILs arrived at 7:57 UTC.

    DNS resolution failuresHigh error rate
    Trigger

    On 21 September 2023, as a planned change in root zone management, a ZONEMD resource record, in effect a checksum of the zone's contents, was added to the root zone for the first time. On 4 October at 07:00 UTC the DNSSEC signatures in the last root zone copy that 1.1.1.1 had loaded expired.

    Root cause
    Software defect
    Component
    Recursive DNS resolver service
    Contributing factor
    Cache-masked failureUnsafe failure behaviourMonitoring blind spotGap in automated tests
    Detection

    External reports of unexpected SERVFAILs started coming in at 7:57 UTC, and Cloudflare declared an internal incident at 8:03 UTC. Nothing had alerted on static_zone serving a stale root zone during the two weeks before.

    Remediation
    • Deployed an override rule meant to stop 1.1.1.1 from answering with the stale root zone, which did not cover forwarded queries
    • Stopped 1.1.1.1 from preloading the root zone file entirely
    Action items
    • Add alerts for when static_zone serves a stale root zone file
    • Re-evaluate how the root zone is ingested and distributed internally so new record types are handled
    • Improve test coverage for what happens when the root zone fails to parse
    • Limit how long stale copies of the root zone can be used, following RFC 8806
    Lessons

    Cloudflare's main lesson is never to assume an input will not change. Modern systems pull in long chains of libraries that may not be ready when their input changes, so they need tests that catch regressions and components that fail gracefully on new input, and format changes in DNS and BGP should be expected to have an impact. Cloudflare also writes that with better monitoring of the stale root zone, the existing caching would have prevented any impact.

    View case
Check your cluster for this

This case involves Recursive DNS resolver service. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
Recursive DNS resolver service
Contributing factor
Cache-masked failureMonitoring blind spotGap in automated testsUnsafe failure behaviour
Organisation
Cloudflare
  1. IL-0340 Cloudflare's 1.1.1.1 resolver returned SERVFAIL to some valid DNS queries for four hours
    Cloudflare2023