Incident Library

IL-0399 Government Digital Service · 2016-12-15awsnetworking

Applications on GOV.UK PaaS in AWS eu-west-1 failed DNS lookups intermittently for about two hours overnight

Many components and operations of GOV.UK PaaS need working DNS resolution. On 15 December 2016 the EC2 DNS resolvers in AWS eu-west-1 had a problem, and from 1.17am VMs in zone 3 of the production account logged DNS resolution errors, Datadog metrics stopped arriving, the Pingdom check of the database endpoint failed several times and CF API smoke tests failed to resolve UAA. AWS acknowledged DNS resolution issues from some instances in EU-WEST-1 on its status page at 1.50am. The DNS errors stopped at 2.57am and AWS reported full recovery at 3.17am. The team was not providing 24/7 support for the PaaS at the time; a team member saw the warnings at 8am, checked monitoring to confirm the issue had ended, and users of the platform were notified by email at 11.06am. Because of the hour, the team thinks tenants were probably affected only through automated scheduled tasks.

The team considered running a local DNS resolver cache instead of querying the EC2 resolvers directly and rejected it, because uncached records would still have failed. It also discussed adding open resolvers such as Google or OpenDNS and decided against them because their rate limiting and other behaviour might introduce new issues; both ideas may be revisited if DNS problems recur.

Severity
–
Scope
Applications on GOV.UK PaaS had intermittent DNS failures and could fail to reach databases, backend services and APIs; some end users would have seen intermittent slowness and errors
Users affected
–
Region
AWS eu-west-1, zone 3
Time to detect
6h 43m
Time to resolve
2h

Symptoms

On 15 December 2016 from 1.17am, VMs of the GOV.UK Platform as a Service (PaaS) in AWS eu-west-1 logged DNS resolution errors, and applications deployed on the PaaS failed DNS queries intermittently. Applications could fail to connect to their databases, backend services and APIs, and some end users of those applications would have seen intermittent slowness and errors between 1am and 3.17am.

The first errors appeared on the VMs cell/4, cell/5, cell/8, consul/2 and etcd/2, all in zone 3 of AWS in the production account. Most of them were failures to resolve the Datadog monitoring endpoint, repeating every 10 seconds, so the Datadog agent could not send metrics. From 1.33am Datadog raised "no-data" alerts from disk monitors on those VMs, followed by 68 alert and recover events from six hosts (cell/4, cell/5, cell/8, consul/2, elasticsearch_master/2, etcd/2) until 02:58.

Pingdom monitoring of the PaaS healthcheck app, whose DB endpoint checks that an application can connect to a backing database, first failed at 1.21am with 3 minutes of downtime, followed by 5 more failures between 1:24 and 2:18 lasting 1 to 6 minutes each. A smoke test of a CF API operation failed at 1.26am because the VM api/1 could not resolve UAA, and 6 more such errors followed between 01:37 and 02:46. The rate of DNS errors dropped at 2.52am and the last one was logged at 2.57am.

Symptom DNS resolution failuresIntermittent connection failuresMonitoring outage

Root cause

Root cause Cloud provider outage

Component VPC DNS resolver
Contributing factor Signal buried in noise
Trigger
–

Detection

Datadog and Pingdom alerts fired during the night, but no one was on out-of-hours support. A PaaS team member noticed the warnings at 8am when starting work and checked monitoring to confirm the issue was over.

Remediation

–

Action items

Lessons

–

Sources

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, VPC DNS resolver among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.