Incident Library

IL-0400 PSBA · 2026-03-25networking

Public sector sites across Wales lost connectivity for over 8 hours as a PE router wrongly advertised a default route

The PSBA network, run by BT PSBA, connects public sector organisations across Wales such as councils, health boards and fire and rescue services. BT/PSBA was taking over the management of legacy UPS devices operated by Capita. The devices were addressed in a /16 subnet, and managing them from BT/PSBA needed a routing change in which MPLS VPN routes were leaked in a controlled way. To see whether this was simple, a route map was applied to a PSBA core PE router without formal change control, because the change was considered non-service impacting. No impact was seen at first. When it became clear that return traffic from the UPS devices would need route import changes as well, the change was removed, since going further needed formal approval. About 20 to 30 minutes later a wider management connectivity problem appeared, and the outage was not linked to the change because the two management links were flapping; the cause of that flapping was still under investigation when the report was written.

The PE router at the Vantage Data Centre where the route map had been changed was exporting a default route, and customer VPNs that still carried legacy MPLS management configuration imported it, overriding the customers' own default routes. With management access unstable, 3rd line could not troubleshoot, and the Service Desk escalated through the Major Incident manager, the account director and a Technical Design Architect, who travelled to Cardiff and then to the Vantage Data Centre. At 22:00 the Technical Design Architect changed QoS on the BGP, management access returned and alarms cleared to single figures by 22:27, but health boards saw no improvement.

Further troubleshooting at the data centre found the exported default route. The route map amendment was confirmed removed, yet the default route was still advertised after routes were cleared and inbound BGP advertisements were reset. At 01:02 the configuration issue affecting the internet learned route was corrected, and at 01:10 the Technical Design Architect did a controlled reload of the router, which restored customer services and management connectivity. The network was stable from 01:15 and health boards confirmed restoration at 02:08. The report also records that the first incident communications at 17:27 went to distribution lists that missed some organisations and that no customer text message was sent with them; the contact list and SMS were widened to all customers across the PSBA estate at 21:48.

Severity
–
Scope
Sites on the PSBA network across Wales lost connectivity or internet access, fully or intermittently, including councils, health boards, fire and rescue services, Natural Resources Wales and others; PSBA management connectivity was also unstable
Users affected
public sector organisations across Wales
Region
Wales
Time to detect
18m
Time to resolve
8h 35m

Symptoms

On 25 March 2026 from about 16:40, public sector organisations on the PSBA network across Wales lost connectivity at their sites. The Service Desk saw about 400 alerts at 16:58, when a Major Incident was declared, about 1,500 at 17:05 and about 5,000 across Wales at 17:08, while councils, a fire and rescue service and Natural Resources Wales phoned in to report sites down.

Customers described intermittent connectivity, complete loss of internet service at some locations, loss of network access and problems signing in. DHCW reported the loss of the default route from the data centre and VPN problems at 17:10, which was first handled as a separate case. Carmarthen had lost default routing at about 17:00. In the PSBA network, a customer VRF with the route target import 64627:16 could have its own default route overridden, which shows up as lost or degraded internet connectivity.

At the same time the engineers lost reliable management access. The core devices were reachable, but management access to both NAT devices kept dropping because BGP peering to the secure management platform was lost, and the two PSBA management links, Cardiff and Bangor, kept flapping. Sessions lasted less than 30 seconds, so 3rd line could only make brief checks, found no faults, and reported that the network looked stable and that troubleshooting had to wait for the management links.

The early investigation did not connect the outage to a routing change made and removed shortly before, because the flapping management links looked like the problem. Circuit checks on the management links and the circuit supplier found no faults, and the team also checked whether a major service outage in the London area was related; it was not.

Symptom Network unreachableIntermittent connection failuresLink flapping

Root cause

Root cause Configuration change

Component BGP routing
Contributing factor Ineffective rollbackMissed in reviewLimited debug accessOverlapping failures
Trigger
At 16:30 a route map change was applied, outside formal change control, on a PSBA core Provider Edge (PE) router to leak MPLS VPN routes, so that legacy Capita-operated UPS devices in a /16 subnet could be managed by BT/PSBA. The change was removed at 16:40.

Detection

At 16:57 Mid Wales Fire & Rescue Service phoned in a loss of connectivity at its Presteigne site, and at 16:58 a case was opened in Service Now for a group of about 400 alerts and a Major Incident was declared.

Remediation

Action items

Lessons

The report names eight lessons: a change was made outside PSBA change control; internal communication during the Major Incident was slow and fragmented; the organisation was not ready for a network-wide incident; losing management access stopped the resolver groups from seeing the network state and customer impact; the resolver groups did not understand customer impact in enough detail, which delayed finding the root cause; escalation to PSBA management was slow; legacy configuration left in the network contributed; and email and SMS communications did not reach some customers in time.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, BGP routing among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.