Cases1 case

1 to 1 of 1

1. IL-0400 Public sector sites across Wales lost connectivity for over 8 hours as a PE router wrongly advertised a default route

   PSBA2026

   Symptoms

   On 25 March 2026 from about 16:40, public sector organisations on the PSBA network across Wales lost connectivity at their sites. The Service Desk saw about 400 alerts at 16:58, when a Major Incident was declared, about 1,500 at 17:05 and about 5,000 across Wales at 17:08, while councils, a fire and rescue service and Natural Resources Wales phoned in to report sites down.

   Network unreachableIntermittent connection failuresLink flapping

   Trigger

   At 16:30 a route map change was applied, outside formal change control, on a PSBA core Provider Edge (PE) router to leak MPLS VPN routes, so that legacy Capita-operated UPS devices in a /16 subnet could be managed by BT/PSBA. The change was removed at 16:40.

   Root cause

   Configuration change

   Component

   BGP routing

   Contributing factor

   Ineffective rollbackMissed in reviewLimited debug accessOverlapping failures

   Detection

   At 16:57 Mid Wales Fire & Rescue Service phoned in a loss of connectivity at its Presteigne site, and at 16:58 a case was opened in Service Now for a group of about 400 alerts and a Major Incident was declared.

   Remediation

   • Removed the route map change from the PE router at 16:40
   • Changed QoS on the BGP to bring back the management links, after which alarms cleared
   • Cleared routes and reset inbound BGP advertisements, which did not stop the default route being advertised
   • Corrected the configuration issue affecting the internet learned route at 01:02
   • Performed a controlled reload of the affected router at 01:10, which restored customer services and management connectivity

   Action items

   • Reinforce change control requirements with an internal briefing on formal change submission, the risks of unplanned changes and roles during planned and emergency changes
   • Create a dedicated Microsoft Teams channel as the single real-time communication point for all live P1 and Major Incidents
   • Explore disaster recovery and business continuity test exercises for major or P1 incidents
   • During loss of management access, have all technical teams assess customer impact from what reporting organisations provide, captured and shared in real time by the Service Desk and Incident Management
   • Investigate a fixed link between Capital Quarter and the PSBA management racks in Vantage Data Centre at Newport, with dedicated management terminals and two-factor authentication for access outside the secure management platform
   • Improve initial triage with stronger probing questions when incidents are reported and keep reassessing and communicating impact throughout the incident
   • Review the P1 and Major Incident process so escalation criteria and timeframes are clearly defined, and train the Service Desk to recognise and escalate them early
   • Review the PSBA estate for legacy configuration, set processes and timeframes for removing it and add controls so it is not introduced again
   • Review email and SMS distribution lists and processes and reinforce Service Desk training on customer communications during P1 and Major Incidents

   Lessons

   The report names eight lessons: a change was made outside PSBA change control; internal communication during the Major Incident was slow and fragmented; the organisation was not ready for a network-wide incident; losing management access stopped the resolver groups from seeing the network state and customer impact; the resolver groups did not understand customer impact in enough detail, which delayed finding the root cause; escalation to PSBA management was slow; legacy configuration left in the network contributed; and email and SMS communications did not reach some customers in time.

   View case

By root cause1