Cases11 cases

4 to 8 of 11

1. IL-0324 Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database cluster

   Wikimedia Foundation2025

   Symptoms

   On 2025-03-12 from 10:58 UTC, alerts for high backend response times came in, and the MediaWiki deployments mw-api-int, mw-parsoid and mw-jobrunner, which serve MediaWiki for non-user-facing purposes, showed high latency and database errors. The External Storage cluster, whose hosts store the compressed text of wiki page revisions, had a sharp rise in load and connections. The same happened again on 2025-03-13 from 11:15 UTC.

   Database overloadHigh latencyHigh error rateLog floodMonitoring outage

   Trigger

   At 10:44 UTC on 12 March a Scap deployment of the PHP 8.1 rollout for mw-api-int, mw-parsoid and mw-jobrunner was run with the flag -Dbuild_mw_container_image:False; the rollout was deployed with Scap again at 11:15 UTC on 13 March.

   Root cause

   Faulty deployment

   Component

   DatabaseIn-memory cacheConfigMaps and SecretsHelmCI system

   Contributing factor

   Unverified assumptionDependence on a warm cacheSignal buried in noise

   Detection

   Alerts for high backend response times started at 10:58 UTC on 12 March, 14 minutes after the Scap deployment; 2 on-call engineers were paged. The MediaWikiMemcachedHighErrorRate alert that pointed at the cause fired only briefly because its log pipeline was overloaded.

   Remediation

   • Deployed a patch to cirrus-streaming-updater that reduced SUP parallelism
   • Reduced the concurrency of the categoryMembershipChange job in changeprop-jobqueue
   • On 12 March, reverted 1126607 and 1126650 and re-ran Scap
   • On 13 March, rolling-restarted changeprop-jobqueue
   • Rolling-restarted mw-api-int
   • Disabled the categorymembership job in changeprop-jobqueue
   • Reverted 1127476 and 1127478
   • Redeployed mw-api-int to pick up the revert
   • Increased the number of mw-api-int replicas
   • Redeployed mw-jobrunner to pick up the revert
   • Redeployed mw-parsoid to pick up the revert

   Action items

   • Reduce the number of messages sent through channel:Memcached during failures
   • Add helm rollback functionality to Scap
   • Decide whether Scap should update helmfile-defaults when -Dbuild_mw_container_image:False is set
   • Consider removing envvars.inc from MediaWiki images
   • Consider "Why" dashboards that give an overview of a group of related components together with relevant links to follow
   • Coordinate incident responders better
   • Be able to work with MediaWiki developers during incidents

   Lessons

   The write-up puts the weight not on the technical root cause but on why the team could not see, even after the 12 March incident, that three MediaWiki deployments were running without Memcached. Alert dashboards showed what was happening, such as many busy PHP workers, but not why; dashboards linking components that affect each other, such as SQLBlobStore keys and External Storage, would have helped. The alert that pointed in the right direction did not stay firing and would probably have been lost among 78 others. The incident coordinator could not tell who was working on what or who had joined or left, the channels were noisy, the external impact was judged more critical than it was, and trace.wikimedia.org was not used although it might have helped.

   View case
2. IL-0280 Kafka brokers in many New Relic cells ran out of disk, leaving US data collection and alerting unusable for hours

   New Relic2021

   Symptoms

   On 29 July 2021, data collection and alerting in New Relic's US region became unusable for many customers for several hours. It began with one unresponsive Kafka broker in a single cell of NRDB, New Relic's telemetry data platform, and over the following hours Kafka brokers in more and more cells failed as their disks filled up.

   Node disk fullThroughput degradationMonitoring outage

   Trigger

   Responders used an incident response tool to extend Kafka data retention for the degraded cell so that its backlog would not be lost. The retention increase was larger than usual, and the tool applied it to all cells of the same type, not only the affected one.

   Root cause

   Operator error

   Component

   Message broker

   Contributing factor

   Harmful manual actionNo staged rolloutSignal buried in noiseUnverified assumption

   Detection

   Engineers saw a Kafka broker in one cell become unresponsive on the morning of 29 July. Later a monitoring alert warned that disk space was running low, but it fired together with unrelated warnings from several other systems and was missed.

   Remediation

   • Restarted the unresponsive Kafka broker by hand, while the automated restart was also running
   • Resolved the disk space problem on the Kafka brokers, after which service recovered slowly

   Lessons

   Cell isolation is the strongest protection against a widespread disruption, and emergency tools have to respect it just as normal operations do. Emergency response tools must be safe to use in chaotic situations without losing their ability to work. Engineers' attention is limited, so incident response processes and alerts need regular review to direct it at the most important signal.

   View case
3. IL-0255 Basecamp's two redundant links between data centers dropped together and a Redis replica resync loop stalled chat

   37signals2020

   Symptoms

   At 11:59AM CDT on Friday 28 August 2020, Basecamp's Ops team received a flood of alerts from PagerDuty, Nagios and Prometheus. Monitoring showed that the newer of the two metro links between the data centers in Ashburn, VA and Chicago, IL had been lost for about 20-30 seconds, and BC3 kept responding slowly after the link came back.

   Link flappingHigh latencyReplication lag

   Trigger

   Both redundant metro links between Basecamp's Ashburn and Chicago data centers dropped for short periods; monitoring showed the newer link down for about 20-30 seconds.

   Root cause

   Dependency failure

   Component

   Backbone network (WAN)BGP routingIn-memory cacheDatabase replication

   Contributing factor

   Single point of failureRetry stormSignal buried in noise

   Detection

   A flood of alerts from PagerDuty, Nagios and Prometheus, with monitoring showing the lost metro link. A lingering alert for one Redis DB instance later pointed to the replication problem.

   Remediation

   • Turned chat off at the load balancer layer so the rest of BC3 could stabilise
   • Restarted Redis on the replica that was stuck in the resynchronization loop

   Action items

   • Make the documented Redis configuration changes for the known replication issue
   • Make concrete plans to remove the shared physical path under the two metro links
   • Prove the failover systems in production at peak load
   • Make explicit room for post-incident follow-up work and push other work out

   Lessons

   Planned redundancy and live testing had given Basecamp a false sense of security, especially at peak load: moving between redundant network links or shifting load across data centers should work without issue, and Basecamp would rather find out in a controlled test with the full team ready. The follow-up from an outage needs urgency and explicitly cleared time, not just a place on a list.

   View case
4. IL-0359 Queries to Gandi LiveDNS timed out when two crashed anycast DNS nodes kept announcing all three name server addresses

   Gandi2020

   Symptoms

   On 16 July 2020 between 13:26 and 13:48 UTC, DNS queries to two nodes of Gandi's LiveDNS platform failed with a timeout, and retries against the other name servers of a domain failed as well. The status page reported impact for FR customers.

   DNS resolution failures

   Trigger

   A software bug in Gandi's DNS server software triggered a fault and stopped the server on two LiveDNS nodes.

   Root cause

   Software defect

   Component

   Authoritative DNS servers and recordsBGP routing

   Contributing factor

   Single point of failureSignal buried in noiseOverlapping failures

   Detection

   Monitoring alerts fired on the DNS servers, but they were first read as false positives among the noise from an earlier internal network incident.

   Action items

   • Fix the way a node is disabled when its DNS server is not working correctly
   • Fix the node configuration and add monitoring so that no node announces several clusters
   • Rework monitoring and internal training and organisation
   • Deploy the fix for the bug in the DNS server software

   View case
5. IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster service

   DFDS2020

   Symptoms

   On 9 June 2020 at 12:25 CET a CloudWatch alert fired on a high count of 5XX responses from the AWS Application Load Balancers (ALBs) of DFDS's Hellman Kubernetes cluster, and at 12:28 the SRE team verified that no ingress traffic was reaching the cluster. Every service running inside the cluster was unreachable from outside it, because both ALBs, the authenticated one and the one without authentication, no longer had any Kubernetes worker nodes attached.

   Service outage

   Trigger

   Planned maintenance to split the Hellman cluster's auto-scaling groups by availability zone applied Terraform changes at 12:20 CET that removed the ALB instance attachments.

   Root cause

   Configuration change

   Component

   TerraformCloud load balancerNode launch template

   Contributing factor

   Unrepresentative stagingSignal buried in noiseMonitoring blind spot

   Detection

   A CloudWatch alert detected a high count of 5XX responses for the ALBs, and the lack of ingress traffic was then verified.

   Remediation

   • Attached the worker node instances to the ALBs by hand
   • Implemented a permanent fix within the hour

   Action items

   • Implement alerts on healthy and unhealthy ALB targets for more precise and immediate notice
   • Reduce alert noise further
   • Implement host name resolution for Goldpinger

   Lessons

   General alert noise is still too high, there was no alert on healthy and unhealthy ALB targets, and test clusters should check ingress through the ALBs.

   View case

By root cause7