Cases2 cases

1 to 2 of 2

1. IL-0268 AdGuard DNS resolvers in Amsterdam, Frankfurt and London overloaded and answered with timeouts and SERVFAIL

   AdGuard DNS2025

   Symptoms

   Between about 15:30 and 18:00 UTC on 25 November 2025 some AdGuard DNS users saw slow or failed DNS resolution, with timeouts and SERVFAIL responses. Alerts fired for increased DNS error rates and server load, and a significant part of the traffic in several European locations, primarily Amsterdam, Frankfurt and London, had intermittent DNS failures for up to 2.5 hours.

   DNS resolution failuresIntermittent timeoutsHigh latencyMemory pressure

   Trigger

   A new version of AdGuard DNS, deployed on 25 November to improve how resolvers cache user data, had a bug that saved only the recently changed subset of user data into the local cache file. The first deployment failed partway on a network issue and left about half the servers on the new version; the second attempt made many resolvers start full synchronizations at once.

   Root cause

   Faulty deployment

   Component

   Recursive DNS resolver serviceLanguage runtime settings

   Contributing factor

   Dependence on a warm cacheRetry stormHarmful manual actionGap in automated tests

   Detection

   Alerts for increased DNS error rates and server load fired during the second deployment attempt, between 15:30 and 16:00 UTC.

   Remediation

   • Rolled back the buggy version on all servers
   • Partially rerouted Amsterdam's traffic to other European locations
   • Reduced connection concurrency and request pipeline depth in Frankfurt and London, from 5 to 1 in Frankfurt
   • Adjusted limits to the actual hardware capacity
   • Repaired corrupted cache files and user data snapshots to stop further full synchronizations
   • Normalised the traffic distribution once the rollback was complete

   Action items

   • Add automated tests that the full user dataset is correctly written to and restored from the local cache file, including detection of empty or truncated caches
   • Add integration tests that simulate resolver startup with an empty or corrupted cache under realistic load
   • Standardize server hardware profiles per location, or keep separate configurations per hardware class
   • Review and adjust connlimit, cache sizes and connection limits for each standard node type
   • Reevaluate and lower pipelining depth where needed
   • Reevaluate how connections and goroutines are limited so resource use stays bounded with decent throughput after a connection break
   • Monitor Go heap size, GC time, GC cycles per second and per-cluster request latency distributions
   • Alert on sustained increases in SERVFAIL rate and request duration while upstreams are healthy
   • Refine DNS resolver deployment and rollback procedures to avoid simultaneous cache invalidation across a large part of the fleet
   • Document and rehearse playbooks for adjusting pipeline depth, connection limits and connlimit during live incidents

   Lessons

   How nodes behaved depended on the interaction between available RAM, Go's garbage collector and the connlimit setting. On 64-GiB nodes a high limit let the heap grow large under heavy concurrency and pipelining, and the garbage collector then spent more and more CPU, a known worst case for high-memory applications in which memory is not exhausted but latency rises and throughput falls. On 32-GiB nodes the same configuration reached practical memory limits sooner because it had not been scaled down. Together with high pipelining and client retries, GC pressure, connection saturation and retries formed a feedback loop.

   View case
2. IL-0285 Clever Cloud's Pulsar brokers stalled in a minor upgrade, then hypervisors overloaded and crashed with kernel panics

   Clever Cloud2024

   Symptoms

   On 2 August 2024, two minutes after Clever Cloud's data team started a minor rolling upgrade of its Apache Pulsar messaging cluster to release 3.3.1, monitoring showed connection problems with the messaging platform. Producing data into the event bus failed, the backlog grew while no new producers or consumers succeeded, and agents on hypervisors, log forwarders on load balancers and deployments were affected. At 10:20 CEST the Pulsar cluster was declared defective.

   Deploys failingSustained high node CPUMemory pressureNode kernel panicOOMKilledAutoscaler misbehaviourMonitoring outage

   Trigger

   At 10:06 CEST the data team began a minor rolling upgrade of Apache Pulsar to release 3.3.1 that also switched on a new balancer for bundle placement: the extensible load manager (loadManagerClassName=org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl) with the TransferShedder load shedding strategy instead of the native OverloadShedder.

   Root cause

   Configuration change

   Component

   Message brokerTelemetry agentLinux kernelAutomated capacity manager

   Contributing factor

   No configuration testsMonitoring blind spotIneffective rollbackUnreviewed defaultsAutomation amplificationRetry stormCircular dependency

   Detection

   Monitoring showed connection issues to the messaging platform at 10:08 CEST, two minutes after the upgrade began, and engineers confirmed at 10:09 that producing data into the event bus failed. Monitoring did not catch the first broker's failure to create the system topic, because that broker stayed up.

   Remediation

   • Rolled back the new unload strategy while keeping Pulsar 3.3.1
   • Shut down the client proxies to release the brokers
   • Switched deployment notification events to RabbitMQ as a workaround
   • Stopped all brokers and started only one, which loaded the conflicting system topic so it could be deleted, then started all brokers
   • Restarted the messaging proxies, after which messages flowed again
   • Stopped the watchdog so it would not reload every failed component, and started components one by one
   • Disabled Metrics API ingestion to protect its storage cluster
   • Stopped the long-term storage for observability metrics (the Warp10/FoundationDB cluster) to free resources
   • Put all monitoring instances in all regions into maintenance so they stopped sending events to the orchestrator
   • Killed the overloaded small VMs one by one
   • Flagged 12 hypervisors to drain workload and shut down some load balancers to free CPU and memory
   • Cleaned zombie VMs to speed up recovery
   • Cleared the monitoring alert queue so queued events would not flood the orchestrator
   • Added parked hypervisors to EU-FR-1 to relieve the overloaded ones
   • Removed an overloaded RabbitMQ node from the cluster used by the orchestrator
   • Rolling-restarted every regional L7 load balancer and every L4 load balancer
   • Restarted all HV agents and the add-on provider APIs
   • Destroyed all dangling VMs stuck in the stopping state
   • Re-enabled Metrics API ingestion with a ramp-up QoS policy

   Action items

   • Track KPIs for Pulsar broker and cluster health, hypervisor load, I/O and kernel panic reboots, zombie VMs, RabbitMQ broker and cluster health and deployment queue consumption, each by time of event, detection, decision and resolution
   • Deal with too many monitoring alerts
   • Look at a backup bastion, because a regional bastion is vulnerable
   • Rework the control plane recovery process
   • Rework the correlation between monitoring and orchestration
   • Revisit fault domain separation against multi-tenancy
   • Rethink runtime images and the role of the hypervisor
   • Isolate waved release upgrades of Pulsar
   • Redesign runtime images with better hypervisor integration
   • Regionalise observability for redundancy
   • Set Vector's buffer.max_events and buffer.max_size for small VMs
   • Switch Vector's buffer.type from memory, the default, to disk
   • Change Vector's buffer.when_full from block, the default, to drop_newest
   • Tune journald verbosity and SystemMaxUse on small VMs
   • Add cgroups restrictions for better process control
   • Add I/O restrictions for the VM manager
   • Detect zombie VMs better

   Lessons

   The simulation process that validates non-minor releases was not applied to minor upgrades or to small configuration changes, and this one went straight to production. Running Clever Cloud on Clever Cloud turned into a chicken-and-egg situation: recovery needed the scheduler to carry out large actions while the control plane was itself slowed and the infrastructure had no free resources. Measuring each key indicator by time of event, detection, decision and resolution showed where the team had lacked visibility or reaction and challenged established rules.

   View case

By root cause2