IL-0268 AdGuard DNS · 2025-11-25networking
AdGuard DNS resolvers in Amsterdam, Frankfurt and London overloaded and answered with timeouts and SERVFAIL
AdGuard DNS resolvers need users' filtering settings locally before they can apply filters. At startup a resolver loads a cached copy of all user data from a local file and then asks the business-logic service only for changes since that snapshot; if the cache is stale or corrupted, it must request the full dataset, which is slow and puts much more load on both sides. A new version deployed on 25 November to improve this saved only the recently changed subset, often very small or empty, into the cache file, so many resolvers started with little or no user data. The first deployment began at 14:30 UTC and failed about an hour later on a network issue, leaving about half the servers on the new version. When the deployment was retried between 15:30 and 16:00, many resolvers found their caches empty or invalid at once and full-synced, which spiked traffic to the business-logic service and resource use on the resolvers.
Alerts fired for DNS error rates and server load, and the investigation found the caching bug. Between 16:00 and 16:30 the team decided to roll back the buggy version on all servers at the same time. Under full syncs and client retries, the mixed Amsterdam cluster failed first on its 32-GiB nodes, whose cache sizes and connection limits had not been scaled down from the 64-GiB settings, and then on the 64-GiB nodes; part of Amsterdam's traffic was rerouted to other European locations, overloading Frankfurt and London. There, connection limits and a pipeline depth of 5 produced many concurrent goroutines and a large heap. AdGuard describes connlimit as the setting that controls how much memory the Go runtime uses for the managed heap before collecting aggressively; set high on 64-GiB nodes, it let the heap grow until the garbage collector used more and more CPU, raising latency. GC pressure, saturated connections and client retries reinforced each other.
From 16:30 to 17:30 configuration changes reduced connection concurrency and pipeline depth in Frankfurt and London and fitted limits to the real hardware, which gradually stabilised the clusters, while corrupted cache files and user data snapshots were repaired to stop further full syncs. Between 17:30 and 18:00 the rollback was completed, caches were consistent, traffic distribution was normalised and error rates were back to baseline.
- Severity
- –
- Scope
- AdGuard DNS resolution in several European locations: intermittent timeouts and SERVFAIL for a significant portion of traffic, for up to 2.5 hours
- Users affected
- some AdGuard DNS users
- Region
- Europe, primarily Amsterdam, Frankfurt and London
- Time to detect
- –
- Time to resolve
- 2h 30m
Symptoms
Between about 15:30 and 18:00 UTC on 25 November 2025 some AdGuard DNS users saw slow or failed DNS resolution, with timeouts and SERVFAIL responses. Alerts fired for increased DNS error rates and server load, and a significant part of the traffic in several European locations, primarily Amsterdam, Frankfurt and London, had intermittent DNS failures for up to 2.5 hours.
The upstream DNS servers kept answering quickly and correctly throughout. The failures came from the AdGuard DNS resolver layer itself, which could not process and return responses before internal or client-side timeouts expired, so request duration and the number of SERVFAIL responses rose sharply.
The trouble started when a retried deployment of a new version made a large number of resolvers find at the same time that their local user-data caches were effectively empty or invalid, and all of them began full synchronizations with the business-logic service. In Amsterdam the cluster mixed machines with 32 GiB and 64 GiB of RAM, while connection limits, cache sizes and memory settings were tuned for 64 GiB. Under the full syncs and a storm of retrying DNS clients, the 32-GiB machines ran out of usable memory and effectively stopped serving traffic; their load moved to the 64-GiB machines, which then hit their connection limits as queues grew. Partly rerouting Amsterdam's traffic to other European locations stabilised Amsterdam but put heavy extra load on Frankfurt and London.
In Frankfurt CPU usage never looked fully saturated at the system level, yet Go garbage collection and internal contention drove up tail latency. The cluster hit its connection limit and stayed there: existing connections were held longer than usual, new clients struggled to connect, and their retries kept the limit saturated. With a request pipeline depth of 5, each connection carried several in-flight queries, each handled by separate goroutines and buffers, which grew the heap and the pressure on the garbage collector. When the pipeline depth was reduced from 5 to 1 in Frankfurt, the system recovered very quickly.
Symptom DNS resolution failuresIntermittent timeoutsHigh latencyMemory pressure
Root cause
Root cause Faulty deployment
- Trigger
- A new version of AdGuard DNS, deployed on 25 November to improve how resolvers cache user data, had a bug that saved only the recently changed subset of user data into the local cache file. The first deployment failed partway on a network issue and left about half the servers on the new version; the second attempt made many resolvers start full synchronizations at once.
Detection
Alerts for increased DNS error rates and server load fired during the second deployment attempt, between 15:30 and 16:00 UTC.
Remediation
- Rolled back the buggy version on all servers
- Partially rerouted Amsterdam's traffic to other European locations
- Reduced connection concurrency and request pipeline depth in Frankfurt and London, from 5 to 1 in Frankfurt
- Adjusted limits to the actual hardware capacity
- Repaired corrupted cache files and user data snapshots to stop further full synchronizations
- Normalised the traffic distribution once the rollback was complete
Action items
- Add automated tests that the full user dataset is correctly written to and restored from the local cache file, including detection of empty or truncated caches
- Add integration tests that simulate resolver startup with an empty or corrupted cache under realistic load
- Standardize server hardware profiles per location, or keep separate configurations per hardware class
- Review and adjust connlimit, cache sizes and connection limits for each standard node type
- Reevaluate and lower pipelining depth where needed
- Reevaluate how connections and goroutines are limited so resource use stays bounded with decent throughput after a connection break
- Monitor Go heap size, GC time, GC cycles per second and per-cluster request latency distributions
- Alert on sustained increases in SERVFAIL rate and request duration while upstreams are healthy
- Refine DNS resolver deployment and rollback procedures to avoid simultaneous cache invalidation across a large part of the fleet
- Document and rehearse playbooks for adjusting pipeline depth, connection limits and connlimit during live incidents
Lessons
How nodes behaved depended on the interaction between available RAM, Go's garbage collector and the connlimit setting. On 64-GiB nodes a high limit let the heap grow large under heavy concurrency and pipelining, and the garbage collector then spent more and more CPU, a known worst case for high-memory applications in which memory is not exhausted but latency rises and throughput falls. On 32-GiB nodes the same configuration reached practical memory limits sooner because it had not been scaled down. Together with high pipelining and client retries, GC pressure, connection saturation and retries formed a feedback loop.
Sources
Related cases
–
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0268
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0268/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.