Cases2 cases

1 to 2 of 2

1. IL-0398 GOV.UK showed "Error 503 Service Unavailable" pages for under an hour during a global outage of its primary CDN

   Government Digital Service2021

   Symptoms

   On 8 June 2021, for less than an hour, people visiting GOV.UK could not reach its information and services and got error pages with the heading "Error 503 Service Unavailable" instead. The outage was in the primary content delivery network (CDN) that GOV.UK uses, and many other websites on the same CDN around the world were down at the same time.

   Service outage

   Root cause

   Dependency failure

   Component

   CDN and edge network

   Contributing factor

   –

   Detection

   The incident was detected 4 minutes after it began, and an incident lead and a communications lead then started the formal incident process.

   Remediation

   • Waited 15 minutes to judge the primary CDN outage, then decided to fail over to the secondary CDN
   • Deployed the prepared DNS record changes, and traffic began to route to the secondary CDN within 30 minutes
   • Switched back to the primary CDN after it recovered, and monitored for an hour before declaring the incident resolved

   Action items

   • Hold a formal incident review of what happened and how the team responded
   • Consider how to improve the speed and transparency of communications
   • Consider how planned changes to platform technology and processes can make recovery from future incidents faster

   Lessons

   The CDN sits at the edge of the network, so when it breaks the whole website goes down, and the team keeps an always-on secondary CDN and a prepared DNS change for that case. Failing over too early can make an incident worse, because the secondary CDN gives a degraded service and primary CDN incidents are often resolved quickly, so the plan waits 15 minutes before failing over.

   View case
2. IL-0399 Applications on GOV.UK PaaS in AWS eu-west-1 failed DNS lookups intermittently for about two hours overnight

   Government Digital Service2016

   Symptoms

   On 15 December 2016 from 1.17am, VMs of the GOV.UK Platform as a Service (PaaS) in AWS eu-west-1 logged DNS resolution errors, and applications deployed on the PaaS failed DNS queries intermittently. Applications could fail to connect to their databases, backend services and APIs, and some end users of those applications would have seen intermittent slowness and errors between 1am and 3.17am.

   DNS resolution failuresIntermittent connection failuresMonitoring outage

   Root cause

   Cloud provider outage

   Component

   VPC DNS resolver

   Contributing factor

   Signal buried in noise

   Detection

   Datadog and Pingdom alerts fired during the night, but no one was on out-of-hours support. A PaaS team member noticed the warnings at 8am when starting work and checked monitoring to confirm the issue was over.

   Action items

   • Stop the monitoring system from sending emails when disk monitoring data is not being received
   • Make notifying tenants about outages easier and quicker
   • Implement alert notifications for out-of-hours support, shaped by this incident
   • Provide 24/7 support for the PaaS in line with the needs of adopting services

   View case

By root cause2