Cases1 case

1 to 1 of 1

1. IL-0399 Applications on GOV.UK PaaS in AWS eu-west-1 failed DNS lookups intermittently for about two hours overnight

   Government Digital Service2016

   Symptoms

   On 15 December 2016 from 1.17am, VMs of the GOV.UK Platform as a Service (PaaS) in AWS eu-west-1 logged DNS resolution errors, and applications deployed on the PaaS failed DNS queries intermittently. Applications could fail to connect to their databases, backend services and APIs, and some end users of those applications would have seen intermittent slowness and errors between 1am and 3.17am.

   DNS resolution failuresIntermittent connection failuresMonitoring outage

   Root cause

   Cloud provider outage

   Component

   VPC DNS resolver

   Contributing factor

   Signal buried in noise

   Detection

   Datadog and Pingdom alerts fired during the night, but no one was on out-of-hours support. A PaaS team member noticed the warnings at 8am when starting work and checked monitoring to confirm the issue was over.

   Action items

   • Stop the monitoring system from sending emails when disk monitoring data is not being received
   • Make notifying tenants about outages easier and quicker
   • Implement alert notifications for out-of-hours support, shaped by this incident
   • Provide 24/7 support for the PaaS in line with the needs of adopting services

   View case

By root cause1