Cases3 cases

1 to 3 of 3

1. IL-0268 AdGuard DNS resolvers in Amsterdam, Frankfurt and London overloaded and answered with timeouts and SERVFAIL

   AdGuard DNS2025

   Symptoms

   Between about 15:30 and 18:00 UTC on 25 November 2025 some AdGuard DNS users saw slow or failed DNS resolution, with timeouts and SERVFAIL responses. Alerts fired for increased DNS error rates and server load, and a significant part of the traffic in several European locations, primarily Amsterdam, Frankfurt and London, had intermittent DNS failures for up to 2.5 hours.

   DNS resolution failuresIntermittent timeoutsHigh latencyMemory pressure

   Trigger

   A new version of AdGuard DNS, deployed on 25 November to improve how resolvers cache user data, had a bug that saved only the recently changed subset of user data into the local cache file. The first deployment failed partway on a network issue and left about half the servers on the new version; the second attempt made many resolvers start full synchronizations at once.

   Root cause

   Faulty deployment

   Component

   Recursive DNS resolver serviceLanguage runtime settings

   Contributing factor

   Dependence on a warm cacheRetry stormHarmful manual actionGap in automated tests

   Detection

   Alerts for increased DNS error rates and server load fired during the second deployment attempt, between 15:30 and 16:00 UTC.

   Remediation

   • Rolled back the buggy version on all servers
   • Partially rerouted Amsterdam's traffic to other European locations
   • Reduced connection concurrency and request pipeline depth in Frankfurt and London, from 5 to 1 in Frankfurt
   • Adjusted limits to the actual hardware capacity
   • Repaired corrupted cache files and user data snapshots to stop further full synchronizations
   • Normalised the traffic distribution once the rollback was complete

   Action items

   • Add automated tests that the full user dataset is correctly written to and restored from the local cache file, including detection of empty or truncated caches
   • Add integration tests that simulate resolver startup with an empty or corrupted cache under realistic load
   • Standardize server hardware profiles per location, or keep separate configurations per hardware class
   • Review and adjust connlimit, cache sizes and connection limits for each standard node type
   • Reevaluate and lower pipelining depth where needed
   • Reevaluate how connections and goroutines are limited so resource use stays bounded with decent throughput after a connection break
   • Monitor Go heap size, GC time, GC cycles per second and per-cluster request latency distributions
   • Alert on sustained increases in SERVFAIL rate and request duration while upstreams are healthy
   • Refine DNS resolver deployment and rollback procedures to avoid simultaneous cache invalidation across a large part of the fleet
   • Document and rehearse playbooks for adjusting pipeline depth, connection limits and connlimit during live incidents

   Lessons

   How nodes behaved depended on the interaction between available RAM, Go's garbage collector and the connlimit setting. On 64-GiB nodes a high limit let the heap grow large under heavy concurrency and pipelining, and the garbage collector then spent more and more CPU, a known worst case for high-memory applications in which memory is not exhausted but latency rises and throughput falls. On 32-GiB nodes the same configuration reached practical memory limits sooner because it had not been scaled down. Together with high pipelining and client retries, GC pressure, connection saturation and retries formed a feedback loop.

   View case
2. IL-0226 Azure Front Door edge servers crashed worldwide on incompatible configuration metadata, causing timeouts and DNS errors

   Microsoft Azure2025

   Symptoms

   From 15:41 UTC on 29 October 2025, customers and Microsoft services using Azure Front Door (AFD) and Azure Content Delivery Network (CDN) got connection timeout errors and DNS resolution issues. Monitoring alerts started the investigation at 15:48 UTC.

   Intermittent timeoutsDNS resolution failuresService outageHigh latency

   Trigger

   At 15:35 UTC a specific sequence of valid customer configuration changes, made across two different AFD control plane build versions, produced incompatible customer configuration metadata that was deployed to the edge sites.

   Root cause

   Software defect

   Component

   CDN and edge networkAuthoritative DNS servers and records

   Contributing factor

   Gap in automated testsChange with a delayed effectIneffective rollbackExcessive blast radius

   Detection

   The configuration protection system detected the data plane impact and blocked configuration changes at 15:43 UTC, and monitoring alerts started the investigation at 15:48 UTC.

   Remediation

   • The configuration protection system automatically blocked all new and in-flight configuration changes worldwide
   • Removed the problematic customer configurations from the latest LKG snapshot by hand
   • Blocked all customer configuration changes from propagating to the data plane
   • Deployed the updated LKG configuration across the global fleet
   • Moved Azure Portal away from Azure Front Door
   • Rebalanced traffic by hand to a few healthy edge sites, then switched back to automatic traffic management
   • Blocked all AFD customer configuration changes at the Azure Resource Manager level until safeguards were in place

   Action items

   • Fix the control plane incompatibility and the data plane bug
   • Process each customer configuration completely and synchronously before production stages
   • Add stages to the phased configuration rollout, with longer bake time between them
   • Move configuration processing on data plane servers into isolated worker processes
   • Reduce configuration propagation time from 45 minutes to about 15 minutes
   • Test backwards compatibility with configurations generated by earlier control plane builds
   • Move critical first-party infrastructure to an active-active setup with fail away
   • Isolate customer configuration and traffic with micro cell segmentation of the AFD data plane
   • Restore customer configurations within one hour by using the local configuration cache
   • Cut data plane recovery further, to about 10 minutes
   • Remove delays in Azure Service Health alerts by raising resource thresholds
   • Extend automated Azure Service Health alerts to similar classes of service degradation
   • Fail over support request systems faster so Premier and Unified customers can always open requests

   Lessons

   Microsoft describes configuration propagation risk as inherent to a global Content Delivery Network, in which route, WAF and origin changes must be deployed quickly worldwide. Both this incident and the 9 October AFD incident came from that risk. This time the incompatible metadata passed a protection system that advances only on healthy data plane signals, because the crash came later from delayed asynchronous processing. Some learnings and repair items from the earlier incident applied here as well. Microsoft advises customers to build a resilient global HTTP ingress layer that stays available during regional or network disruptions, to manage resources through the REST API or PowerShell when management portals are unavailable, and to configure Azure Service Health alerts so the right people hear about service issues.

   View case
3. IL-0340 Cloudflare's 1.1.1.1 resolver returned SERVFAIL to some valid DNS queries for four hours

   Cloudflare2023

   Symptoms

   From 07:00 to 11:00 UTC on 4 October 2023, some users of Cloudflare's 1.1.1.1 DNS resolver received SERVFAIL responses to valid queries. This reached users of WARP and Zero Trust and of third-party DNS resolvers that use 1.1.1.1, and the first external reports of unexpected SERVFAILs arrived at 7:57 UTC.

   DNS resolution failuresHigh error rate

   Trigger

   On 21 September 2023, as a planned change in root zone management, a ZONEMD resource record, in effect a checksum of the zone's contents, was added to the root zone for the first time. On 4 October at 07:00 UTC the DNSSEC signatures in the last root zone copy that 1.1.1.1 had loaded expired.

   Root cause

   Software defect

   Component

   Recursive DNS resolver service

   Contributing factor

   Cache-masked failureUnsafe failure behaviourMonitoring blind spotGap in automated tests

   Detection

   External reports of unexpected SERVFAILs started coming in at 7:57 UTC, and Cloudflare declared an internal incident at 8:03 UTC. Nothing had alerted on static_zone serving a stale root zone during the two weeks before.

   Remediation

   • Deployed an override rule meant to stop 1.1.1.1 from answering with the stale root zone, which did not cover forwarded queries
   • Stopped 1.1.1.1 from preloading the root zone file entirely

   Action items

   • Add alerts for when static_zone serves a stale root zone file
   • Re-evaluate how the root zone is ingested and distributed internally so new record types are handled
   • Improve test coverage for what happens when the root zone fails to parse
   • Limit how long stale copies of the root zone can be used, following RFC 8806

   Lessons

   Cloudflare's main lesson is never to assume an input will not change. Modern systems pull in long chains of libraries that may not be ready when their input changes, so they need tests that catch regressions and components that fail gracefully on new input, and format changes in DNS and BGP should be expected to have an impact. Cloudflare also writes that with better monitoring of the stale root zone, the existing caching would have prevented any impact.

   View case

By root cause2