Cases11 cases

6 to 10 of 11

1. IL-0255 Basecamp's two redundant links between data centers dropped together and a Redis replica resync loop stalled chat

   37signals2020

   Symptoms

   At 11:59AM CDT on Friday 28 August 2020, Basecamp's Ops team received a flood of alerts from PagerDuty, Nagios and Prometheus. Monitoring showed that the newer of the two metro links between the data centers in Ashburn, VA and Chicago, IL had been lost for about 20-30 seconds, and BC3 kept responding slowly after the link came back.

   Link flappingHigh latencyReplication lag

   Trigger

   Both redundant metro links between Basecamp's Ashburn and Chicago data centers dropped for short periods; monitoring showed the newer link down for about 20-30 seconds.

   Root cause

   Dependency failure

   Component

   Backbone network (WAN)BGP routingIn-memory cacheDatabase replication

   Contributing factor

   Single point of failureRetry stormSignal buried in noise

   Detection

   A flood of alerts from PagerDuty, Nagios and Prometheus, with monitoring showing the lost metro link. A lingering alert for one Redis DB instance later pointed to the replication problem.

   Remediation

   • Turned chat off at the load balancer layer so the rest of BC3 could stabilise
   • Restarted Redis on the replica that was stuck in the resynchronization loop

   Action items

   • Make the documented Redis configuration changes for the known replication issue
   • Make concrete plans to remove the shared physical path under the two metro links
   • Prove the failover systems in production at peak load
   • Make explicit room for post-incident follow-up work and push other work out

   Lessons

   Planned redundancy and live testing had given Basecamp a false sense of security, especially at peak load: moving between redundant network links or shifting load across data centers should work without issue, and Basecamp would rather find out in a controlled test with the full team ready. The follow-up from an outage needs urgency and explicitly cleared time, not just a place on a list.

   View case
2. IL-0359 Queries to Gandi LiveDNS timed out when two crashed anycast DNS nodes kept announcing all three name server addresses

   Gandi2020

   Symptoms

   On 16 July 2020 between 13:26 and 13:48 UTC, DNS queries to two nodes of Gandi's LiveDNS platform failed with a timeout, and retries against the other name servers of a domain failed as well. The status page reported impact for FR customers.

   DNS resolution failures

   Trigger

   A software bug in Gandi's DNS server software triggered a fault and stopped the server on two LiveDNS nodes.

   Root cause

   Software defect

   Component

   Authoritative DNS servers and recordsBGP routing

   Contributing factor

   Single point of failureSignal buried in noiseOverlapping failures

   Detection

   Monitoring alerts fired on the DNS servers, but they were first read as false positives among the noise from an earlier internal network incident.

   Action items

   • Fix the way a node is disabled when its DNS server is not working correctly
   • Fix the node configuration and add monitoring so that no node announces several clusters
   • Rework monitoring and internal training and organisation
   • Deploy the fix for the bug in the DNS server software

   View case
3. IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster service

   DFDS2020

   Symptoms

   On 9 June 2020 at 12:25 CET a CloudWatch alert fired on a high count of 5XX responses from the AWS Application Load Balancers (ALBs) of DFDS's Hellman Kubernetes cluster, and at 12:28 the SRE team verified that no ingress traffic was reaching the cluster. Every service running inside the cluster was unreachable from outside it, because both ALBs, the authenticated one and the one without authentication, no longer had any Kubernetes worker nodes attached.

   Service outage

   Trigger

   Planned maintenance to split the Hellman cluster's auto-scaling groups by availability zone applied Terraform changes at 12:20 CET that removed the ALB instance attachments.

   Root cause

   Configuration change

   Component

   TerraformCloud load balancerNode launch template

   Contributing factor

   Unrepresentative stagingSignal buried in noiseMonitoring blind spot

   Detection

   A CloudWatch alert detected a high count of 5XX responses for the ALBs, and the lack of ingress traffic was then verified.

   Remediation

   • Attached the worker node instances to the ALBs by hand
   • Implemented a permanent fix within the hour

   Action items

   • Implement alerts on healthy and unhealthy ALB targets for more precise and immediate notice
   • Reduce alert noise further
   • Implement host name resolution for Goldpinger

   Lessons

   General alert noise is still too high, there was no alert on healthy and unhealthy ALB targets, and test clusters should check ingress through the ALBs.

   View case
4. IL-0211 A Mandrill PostgreSQL shard shut down at transaction ID wraparound and email sending was impaired for about 40 hours

   Mailchimp2019

   Symptoms

   From 05:35 UTC on 4 February 2019 Mandrill, Mailchimp's transactional email service, sent only 80% of queued emails, because one of its PostgreSQL shards, shard4, stopped accepting writes. When Mandrill engineers opened an investigation at 11:58 UTC, they saw a sharply elevated load on shard4.

   Writes rejectedThroughput degradationLoad imbalanceNode disk full

   Trigger

   At 05:35 UTC on 4 February the transaction ID counter of shard4, the busiest Mandrill PostgreSQL shard, reached its upper limit and PostgreSQL shut the database down to protect against wraparound.

   Root cause

   Capacity exhaustion

   Component

   DatabasePostgreSQL vacuum

   Contributing factor

   Ignored early signalNo alert on limitSignal buried in noiseRetry stormMonitoring blind spot

   Detection

   The first exception was logged overnight but was buried by unrelated exceptions in the centralized error reporting. Mandrill engineers opened an investigation at 11:58 UTC and, while looking for the source of the rising load on shard4, found the wraparound at 12:56 UTC.

   Remediation

   • Put shard4 into standalone mode and started VACUUM at 14:16 UTC on 4 February
   • Stopped and restarted the vacuum with settings that made it run faster
   • Started a dump and restore of the database without the Search and Url tables while the vacuum ran
   • Replaced storage volumes on the Mandrill app servers whose disks were filling with queued jobs and logs
   • Moved the job-queue tables from shard4 to other live shards
   • Truncated the Search and Url tables on the locked database at 19:12 UTC on 5 February, after which the vacuum finished in about an hour
   • Started sending from the backlog at 21:36 UTC
   • Added alerting for XID wraparound by 22:40 UTC

   Lessons

   Mailchimp named the factors that slowed its response: knowledge of and access to Mandrill systems and monitoring rested with a few people, Mandrill logging needed a locally writable disk, logs of malformed API requests were hard to separate from real application errors, job health was hard to see, and the disk space problem made partial deployments for triage risky. What helped was trained incident commanders with clear roles (incident commanders in 3-hour rotations, scribes, technical liaisons and points of contact between teams), periodic summaries with a list of recent and open decisions, and taking time in a video meeting to reassess, which led to the truncation. Running in the cloud let the team expand storage quickly, and the team was lucky that the largest table holding up the vacuum held data it could delete.

   View case
5. IL-0399 Applications on GOV.UK PaaS in AWS eu-west-1 failed DNS lookups intermittently for about two hours overnight

   Government Digital Service2016

   Symptoms

   On 15 December 2016 from 1.17am, VMs of the GOV.UK Platform as a Service (PaaS) in AWS eu-west-1 logged DNS resolution errors, and applications deployed on the PaaS failed DNS queries intermittently. Applications could fail to connect to their databases, backend services and APIs, and some end users of those applications would have seen intermittent slowness and errors between 1am and 3.17am.

   DNS resolution failuresIntermittent connection failuresMonitoring outage

   Root cause

   Cloud provider outage

   Component

   VPC DNS resolver

   Contributing factor

   Signal buried in noise

   Detection

   Datadog and Pingdom alerts fired during the night, but no one was on out-of-hours support. A PaaS team member noticed the warnings at 8am when starting work and checked monitoring to confirm the issue was over.

   Action items

   • Stop the monitoring system from sending emails when disk monitoring data is not being received
   • Make notifying tenants about outages easier and quicker
   • Implement alert notifications for out-of-hours support, shaped by this incident
   • Provide 24/7 support for the PaaS in line with the needs of adopting services

   View case

By root cause7