Incident Library

IL-0398 Government Digital Service · 2021-06-08networking

GOV.UK showed "Error 503 Service Unavailable" pages for under an hour during a global outage of its primary CDN

GOV.UK serves roughly 93% of its requests from the cache of its primary CDN. The team treats the CDN as a possible single point of failure and keeps a secondary CDN always on but normally without traffic. Moving traffic to it, which the team calls failing over, is done by changing GOV.UK's DNS records, and the code for that change was ready to run. Because the secondary CDN gives a degraded service in which dynamic parts such as search and location based services do not work as well, and because primary CDN incidents are often short, the plan was to wait 15 minutes to judge the severity before failing over.

On 8 June 2021 the primary CDN had an outage and GOV.UK showed error pages. The incident was detected 4 minutes after it began, an incident lead and a communications lead started the formal incident process, and the incident team confirmed the CDN as the cause and found the documented failover process. After 15 minutes with no change in the CDN's status the team decided to fail over. DNS changes take time to take effect; within 30 minutes the changes were deployed and traffic began to move to the secondary CDN. By then the primary CDN appeared to be recovering, so after watching it for a few minutes the team switched back to it, monitored for an hour and declared the incident resolved.

Severity
–
Scope
GOV.UK information and services were unreachable and showed error pages for less than an hour; other websites on the same CDN were also affected
Users affected
all visitors to GOV.UK
Region
worldwide
Time to detect
4m
Time to resolve
–

Symptoms

On 8 June 2021, for less than an hour, people visiting GOV.UK could not reach its information and services and got error pages with the heading "Error 503 Service Unavailable" instead. The outage was in the primary content delivery network (CDN) that GOV.UK uses, and many other websites on the same CDN around the world were down at the same time.

The error page was outside the GOV.UK team's control. GOV.UK's CDN is set up to serve static fallback pages, called mirrors, when something goes wrong with GOV.UK's main infrastructure behind it. The CDN sits at the edge of the network, so when it breaks the whole website goes down.

Symptom Service outage

Root cause

Root cause Dependency failure

Component CDN and edge network
Contributing factor –
Trigger
–

Detection

The incident was detected 4 minutes after it began, and an incident lead and a communications lead then started the formal incident process.

Remediation

Action items

Lessons

The CDN sits at the edge of the network, so when it breaks the whole website goes down, and the team keeps an always-on secondary CDN and a prepared DNS change for that case. Failing over too early can make an incident worse, because the secondary CDN gives a degraded service and primary CDN incidents are often resolved quickly, so the plan waits 15 minutes before failing over.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, CDN and edge network among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.