Cases2 cases

1 to 2 of 2

1. IL-0359 Queries to Gandi LiveDNS timed out when two crashed anycast DNS nodes kept announcing all three name server addresses

   Gandi2020

   Symptoms

   On 16 July 2020 between 13:26 and 13:48 UTC, DNS queries to two nodes of Gandi's LiveDNS platform failed with a timeout, and retries against the other name servers of a domain failed as well. The status page reported impact for FR customers.

   DNS resolution failures

   Trigger

   A software bug in Gandi's DNS server software triggered a fault and stopped the server on two LiveDNS nodes.

   Root cause

   Software defect

   Component

   Authoritative DNS servers and recordsBGP routing

   Contributing factor

   Single point of failureSignal buried in noiseOverlapping failures

   Detection

   Monitoring alerts fired on the DNS servers, but they were first read as false positives among the noise from an earlier internal network incident.

   Action items

   • Fix the way a node is disabled when its DNS server is not working correctly
   • Fix the node configuration and add monitoring so that no node announces several clusters
   • Rework monitoring and internal training and organisation
   • Deploy the fix for the bug in the DNS server software

   View case
2. IL-0399 Applications on GOV.UK PaaS in AWS eu-west-1 failed DNS lookups intermittently for about two hours overnight

   Government Digital Service2016

   Symptoms

   On 15 December 2016 from 1.17am, VMs of the GOV.UK Platform as a Service (PaaS) in AWS eu-west-1 logged DNS resolution errors, and applications deployed on the PaaS failed DNS queries intermittently. Applications could fail to connect to their databases, backend services and APIs, and some end users of those applications would have seen intermittent slowness and errors between 1am and 3.17am.

   DNS resolution failuresIntermittent connection failuresMonitoring outage

   Root cause

   Cloud provider outage

   Component

   VPC DNS resolver

   Contributing factor

   Signal buried in noise

   Detection

   Datadog and Pingdom alerts fired during the night, but no one was on out-of-hours support. A PaaS team member noticed the warnings at 8am when starting work and checked monitoring to confirm the issue was over.

   Action items

   • Stop the monitoring system from sending emails when disk monitoring data is not being received
   • Make notifying tenants about outages easier and quicker
   • Implement alert notifications for out-of-hours support, shaped by this incident
   • Provide 24/7 support for the PaaS in line with the needs of adopting services

   View case

By root cause2