Cases1 case
1 to 1 of 1
- Symptoms
On 8 June 2021, for less than an hour, people visiting GOV.UK could not reach its information and services and got error pages with the heading "Error 503 Service Unavailable" instead. The outage was in the primary content delivery network (CDN) that GOV.UK uses, and many other websites on the same CDN around the world were down at the same time.
- Root cause
- Component
- Contributing factor
- –
- Detection
The incident was detected 4 minutes after it began, and an incident lead and a communications lead then started the formal incident process.
- Remediation
- Waited 15 minutes to judge the primary CDN outage, then decided to fail over to the secondary CDN
- Deployed the prepared DNS record changes, and traffic began to route to the secondary CDN within 30 minutes
- Switched back to the primary CDN after it recovered, and monitored for an hour before declaring the incident resolved
- Action items
- Hold a formal incident review of what happened and how the team responded
- Consider how to improve the speed and transparency of communications
- Consider how planned changes to platform technology and processes can make recovery from future incidents faster
- Lessons
The CDN sits at the edge of the network, so when it breaks the whole website goes down, and the team keeps an always-on secondary CDN and a prepared DNS change for that case. Failing over too early can make an incident worse, because the secondary CDN gives a degraded service and primary CDN incidents are often resolved quickly, so the plan waits 15 minutes before failing over.
This case involves CDN and edge network. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.