Cases7 cases

1 to 5 of 7

1. IL-0268 AdGuard DNS resolvers in Amsterdam, Frankfurt and London overloaded and answered with timeouts and SERVFAIL

   AdGuard DNS2025

   Symptoms

   Between about 15:30 and 18:00 UTC on 25 November 2025 some AdGuard DNS users saw slow or failed DNS resolution, with timeouts and SERVFAIL responses. Alerts fired for increased DNS error rates and server load, and a significant part of the traffic in several European locations, primarily Amsterdam, Frankfurt and London, had intermittent DNS failures for up to 2.5 hours.

   DNS resolution failuresIntermittent timeoutsHigh latencyMemory pressure

   Trigger

   A new version of AdGuard DNS, deployed on 25 November to improve how resolvers cache user data, had a bug that saved only the recently changed subset of user data into the local cache file. The first deployment failed partway on a network issue and left about half the servers on the new version; the second attempt made many resolvers start full synchronizations at once.

   Root cause

   Faulty deployment

   Component

   Recursive DNS resolver serviceLanguage runtime settings

   Contributing factor

   Dependence on a warm cacheRetry stormHarmful manual actionGap in automated tests

   Detection

   Alerts for increased DNS error rates and server load fired during the second deployment attempt, between 15:30 and 16:00 UTC.

   Remediation

   • Rolled back the buggy version on all servers
   • Partially rerouted Amsterdam's traffic to other European locations
   • Reduced connection concurrency and request pipeline depth in Frankfurt and London, from 5 to 1 in Frankfurt
   • Adjusted limits to the actual hardware capacity
   • Repaired corrupted cache files and user data snapshots to stop further full synchronizations
   • Normalised the traffic distribution once the rollback was complete

   Action items

   • Add automated tests that the full user dataset is correctly written to and restored from the local cache file, including detection of empty or truncated caches
   • Add integration tests that simulate resolver startup with an empty or corrupted cache under realistic load
   • Standardize server hardware profiles per location, or keep separate configurations per hardware class
   • Review and adjust connlimit, cache sizes and connection limits for each standard node type
   • Reevaluate and lower pipelining depth where needed
   • Reevaluate how connections and goroutines are limited so resource use stays bounded with decent throughput after a connection break
   • Monitor Go heap size, GC time, GC cycles per second and per-cluster request latency distributions
   • Alert on sustained increases in SERVFAIL rate and request duration while upstreams are healthy
   • Refine DNS resolver deployment and rollback procedures to avoid simultaneous cache invalidation across a large part of the fleet
   • Document and rehearse playbooks for adjusting pipeline depth, connection limits and connlimit during live incidents

   Lessons

   How nodes behaved depended on the interaction between available RAM, Go's garbage collector and the connlimit setting. On 64-GiB nodes a high limit let the heap grow large under heavy concurrency and pipelining, and the garbage collector then spent more and more CPU, a known worst case for high-memory applications in which memory is not exhausted but latency rises and throughput falls. On 32-GiB nodes the same configuration reached practical memory limits sooner because it had not been scaled down. Together with high pipelining and client retries, GC pressure, connection saturation and retries formed a feedback loop.

   View case
2. IL-0324 Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database cluster

   Wikimedia Foundation2025

   Symptoms

   On 2025-03-12 from 10:58 UTC, alerts for high backend response times came in, and the MediaWiki deployments mw-api-int, mw-parsoid and mw-jobrunner, which serve MediaWiki for non-user-facing purposes, showed high latency and database errors. The External Storage cluster, whose hosts store the compressed text of wiki page revisions, had a sharp rise in load and connections. The same happened again on 2025-03-13 from 11:15 UTC.

   Database overloadHigh latencyHigh error rateLog floodMonitoring outage

   Trigger

   At 10:44 UTC on 12 March a Scap deployment of the PHP 8.1 rollout for mw-api-int, mw-parsoid and mw-jobrunner was run with the flag -Dbuild_mw_container_image:False; the rollout was deployed with Scap again at 11:15 UTC on 13 March.

   Root cause

   Faulty deployment

   Component

   DatabaseIn-memory cacheConfigMaps and SecretsHelmCI system

   Contributing factor

   Unverified assumptionDependence on a warm cacheSignal buried in noise

   Detection

   Alerts for high backend response times started at 10:58 UTC on 12 March, 14 minutes after the Scap deployment; 2 on-call engineers were paged. The MediaWikiMemcachedHighErrorRate alert that pointed at the cause fired only briefly because its log pipeline was overloaded.

   Remediation

   • Deployed a patch to cirrus-streaming-updater that reduced SUP parallelism
   • Reduced the concurrency of the categoryMembershipChange job in changeprop-jobqueue
   • On 12 March, reverted 1126607 and 1126650 and re-ran Scap
   • On 13 March, rolling-restarted changeprop-jobqueue
   • Rolling-restarted mw-api-int
   • Disabled the categorymembership job in changeprop-jobqueue
   • Reverted 1127476 and 1127478
   • Redeployed mw-api-int to pick up the revert
   • Increased the number of mw-api-int replicas
   • Redeployed mw-jobrunner to pick up the revert
   • Redeployed mw-parsoid to pick up the revert

   Action items

   • Reduce the number of messages sent through channel:Memcached during failures
   • Add helm rollback functionality to Scap
   • Decide whether Scap should update helmfile-defaults when -Dbuild_mw_container_image:False is set
   • Consider removing envvars.inc from MediaWiki images
   • Consider "Why" dashboards that give an overview of a group of related components together with relevant links to follow
   • Coordinate incident responders better
   • Be able to work with MediaWiki developers during incidents

   Lessons

   The write-up puts the weight not on the technical root cause but on why the team could not see, even after the 12 March incident, that three MediaWiki deployments were running without Memcached. Alert dashboards showed what was happening, such as many busy PHP workers, but not why; dashboards linking components that affect each other, such as SQLBlobStore keys and External Storage, would have helped. The alert that pointed in the right direction did not stay firing and would probably have been lost among 78 others. The incident coordinator could not tell who was working on what or who had joined or left, the channels were noisy, the external impact was judged more critical than it was, and trace.wikimedia.org was not used although it might have helped.

   View case
3. IL-0270 Discord's ScyllaDB authentication cluster overloaded during an OS upgrade, and logins failed for about 50 minutes

   Discord2023

   Symptoms

   On 6 November 2023 at 10:52 AM PST, when an upgrade script drained all connections from zone [c] of the scylla-authentication ScyllaDB cluster, Discord's availability, measured as the share of HTTP requests that return a 200-level response, fell at once to about 25%. Users could not connect, log in, read or send messages.

   Service outageDatabase overloadHigh latency

   Trigger

   During an OS upgrade that takes one zone of a ScyllaDB cluster offline at a time, the upgrade script drained all connections from zone [c] of scylla-authentication at 10:52 AM, while one node in the just-upgraded zone [b] had a degraded NVMe Local SSD.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No headroomOverlapping failuresDependence on a warm cacheUnverified assumptionIgnored early signalRetry storm

   Detection

   An incident response call had already started when the first alerts fired at 10:56 AM, four minutes after the start. The time spent by downstream service graph showed the authentication service spending far more time on user requests than normal, and the authentication alerts fired moments later.

   Remediation

   • Turned off sending typing events and marking messages as read to shed load from the authentication cluster
   • Waited for the zone [c] RAID rebuild instead of restarting nodes, and the two zone [c] nodes came back online at 11:41 AM
   • Restarted the two zone [b] nodes one at a time

   Action items

   • Double the size of the scylla-authentication cluster so each node takes half as many requests
   • Rebuild scylla-authentication-b-2 with new, non-degraded disks
   • Build a way to detect disk degradation that is too subtle to affect overall latency but still threatens reliability
   • Move upgrade operations into a more automatic control plane for the Scylla clusters that analyses cluster state, such as cache hit rate
   • Increase the redundancy of the scylla-authentication cluster; the options being explored, not yet decided, are a higher replication factor or a hot-standby cluster in another region

   Lessons

   Taking a whole zone out of a three-zone cluster leaves no margin, so one additional slow node in another zone was enough to overload it. For some Scylla clusters the row cache hit rate of restarted nodes must reach a safe level before the next zone goes down, and a short zonal test beforehand gave false confidence. The small but critical authentication cluster needs to be run differently from Discord's much larger Scylla clusters, with more redundancy.

   View case
4. IL-0216 Wikimedia on-wiki search was slow or rejected after a datacenter switchover overloaded one Elasticsearch node in codfw

   Wikimedia Foundation2023

   Symptoms

   Right after the primary datacenter was switched from EQIAD to CODFW at 14:00 UTC on 20 September 2023, on-wiki search on all Wikimedia wikis became delayed or unavailable for a subset of users. At 14:08 the alert "CirrusSearch codfw 95th percentile latency - more_like" fired and reached IRC and the Search Platform email list as "PROBLEM alert - graphite1005/CirrusSearch codfw 95th percentile latency - more_like is CRITICAL".

   High latencyHigh error rateLoad imbalanceMisrouted requests

   Trigger

   At 14:00 UTC the primary datacenter was switched from EQIAD to CODFW. more_like queries that were expected to go to eqiad, the non-primary datacenter, went to codfw instead, apparently because RESTBase sent them as POST rather than GET.

   Root cause

   Software defect

   Component

   Elasticsearch or OpenSearchIn-memory cache

   Contributing factor

   Unverified assumptionDependence on a warm cacheNo runbook

   Detection

   Automated monitoring caught the problem, but not quickly enough to prevent user-facing impact: the Grafana alert "CirrusSearch codfw 95th percentile latency - more_like" went to IRC and the Search Platform email list at 14:08. No one was paged; SREs were already watching closely because of the datacenter switchover. At 14:14 Hashar pinged #wikimedia-search and engineers began troubleshooting.

   Remediation

   • Banned the overloaded node elastic2044 from the Elasticsearch cluster

   Action items

   • Work out which Elasticsearch steps are needed before a datacenter switchover
   • Update the Elasticsearch section of the datacenter switchover documentation
   • Review the suggested switchover improvements in T285347 and decide whether to implement them
   • Make the RESTBase /related/{article} endpoint send GET instead of POST so the CirrusSearch query cache stays warm in both datacenters
   • Review the shard balance of busy wikis, especially enwiki_content

   Lessons

   Monitoring caught the problem, but not quickly enough to prevent user-facing impact. The heightened attention during the switchover meant the impact was recognised immediately, and banning a single node was quick and easy. The mitigation was not guided by an existing runbook, and the shards on the overloaded host were not checked before it was banned.

   View case
5. IL-0217 Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outage

   Honeycomb2023

   Symptoms

   Hours after a routine switch between two clusters of Retriever, Honeycomb's storage and query engine, late on 24 July 2023, the internal SLO for Shepherd, the ingest service, began to burn slowly. The performance was only marginally slow and inconsistent, with no good explanation, so engineers treated it as minor and left it for the morning.

   High latencyDatabase overloadLock contention or deadlockService outage

   Trigger

   Late on 24 July Honeycomb switched between two Retriever clusters to avoid a subtle bug, and the new cluster stopped writing the last-written timestamps the schema cache depends on. On the morning of 25 July a feature flag flip meant to move those writes back stopped the remaining writes entirely.

   Root cause

   Software defect

   Component

   DatabaseIn-memory cache

   Contributing factor

   Dependence on a warm cacheIgnored early signalHarmful manual actionUnverified assumption

   Detection

   Hours after the cluster switch, the internal SLO for the ingest service Shepherd started burning slowly. Engineers left it for the morning, when a chart showed that Retriever's database calls updating the last-written records had stopped.

   Remediation

   • Flipped only the writes back to the other Retriever cluster with a feature flag, which did not fix the problem
   • Set up circuit breakers that denied all Shepherd traffic with a 5xx error to protect the database
   • Failed the hard-locked database over to a replica
   • Restarted all Retriever hosts so they could provide data for the cache
   • Set the last-written timestamp of every schema with traffic in the last day to now, so the schema store reloaded the cache
   • Removed the circuit breaker and restarted the query hosts that failed internal checks

   Action items

   • Finish the Retriever cluster migration and remove all code that could disable writes behind the cache
   • Explore ways to strengthen the cache further
   • Lower the contention put on the database during update storms from new schemas
   • Stabilise the performance costs of some operations in the system
   • Study instrumentation and experimentation that detect and handle such edge cases better

   Lessons

   Locally good decisions combined into the outage: the query engine wrote the timestamps because it already saw all the data, the shared cache helped scale ingest and untangle the main database, the cluster switch was done to avoid a bug, frequent deploys hid the feature flag bug, pausing deploys during the investigation kept hosts from restarting, and past near-misses made the team expect a slow failover. Feature flags, frequent deploys, pausing deploys during incidents and learning from near-misses all contributed, although they are among the practices Honeycomb trusts most. For a rare lockup of this kind a database failover is the fastest recovery. Several recent Honeycomb incidents started from trying to avoid a subtle bug.

   View case

By root cause3