Graph
Cases5 cases
1 to 5 of 5
IL-0400Public sector sites across Wales lost connectivity for over 8 hours as a PE router wrongly advertised a default routePSBA2026- Symptoms
On 25 March 2026 from about 16:40, public sector organisations on the PSBA network across Wales lost connectivity at their sites. The Service Desk saw about 400 alerts at 16:58, when a Major Incident was declared, about 1,500 at 17:05 and about 5,000 across Wales at 17:08, while councils, a fire and rescue service and Natural Resources Wales phoned in to report sites down.
- Trigger
At 16:30 a route map change was applied, outside formal change control, on a PSBA core Provider Edge (PE) router to leak MPLS VPN routes, so that legacy Capita-operated UPS devices in a /16 subnet could be managed by BT/PSBA. The change was removed at 16:40.
- Root cause
- Component
- Contributing factor
- Detection
At 16:57 Mid Wales Fire & Rescue Service phoned in a loss of connectivity at its Presteigne site, and at 16:58 a case was opened in Service Now for a group of about 400 alerts and a Major Incident was declared.
- Remediation
- Removed the route map change from the PE router at 16:40
- Changed QoS on the BGP to bring back the management links, after which alarms cleared
- Cleared routes and reset inbound BGP advertisements, which did not stop the default route being advertised
- Corrected the configuration issue affecting the internet learned route at 01:02
- Performed a controlled reload of the affected router at 01:10, which restored customer services and management connectivity
- Action items
- Reinforce change control requirements with an internal briefing on formal change submission, the risks of unplanned changes and roles during planned and emergency changes
- Create a dedicated Microsoft Teams channel as the single real-time communication point for all live P1 and Major Incidents
- Explore disaster recovery and business continuity test exercises for major or P1 incidents
- During loss of management access, have all technical teams assess customer impact from what reporting organisations provide, captured and shared in real time by the Service Desk and Incident Management
- Investigate a fixed link between Capital Quarter and the PSBA management racks in Vantage Data Centre at Newport, with dedicated management terminals and two-factor authentication for access outside the secure management platform
- Improve initial triage with stronger probing questions when incidents are reported and keep reassessing and communicating impact throughout the incident
- Review the P1 and Major Incident process so escalation criteria and timeframes are clearly defined, and train the Service Desk to recognise and escalate them early
- Review the PSBA estate for legacy configuration, set processes and timeframes for removing it and add controls so it is not introduced again
- Review email and SMS distribution lists and processes and reinforce Service Desk training on customer communications during P1 and Major Incidents
- Lessons
The report names eight lessons: a change was made outside PSBA change control; internal communication during the Major Incident was slow and fragmented; the organisation was not ready for a network-wide incident; losing management access stopped the resolver groups from seeing the network state and customer impact; the resolver groups did not understand customer impact in enough detail, which delayed finding the root cause; escalation to PSBA management was slow; legacy configuration left in the network contributed; and email and SMS communications did not reach some customers in time.
IL-0509SourceHut was down for 170 hours after a DDoS led its upstream network to null-route the PHL datacenter's subnetSourceHut2024- Symptoms
At about 06:00 UTC on 10 January 2024, SourceHut's monitoring raised the alarm as a layer 3 DDoS attack hit its infrastructure in the PHL datacenter, and at about 09:00 UTC SourceHut lost all access to its PHL network. PHL was the primary datacenter, and the network became unreachable for SourceHut staff and the general public alike.
- Trigger
A layer 3 DDoS attack against SourceHut's PHL infrastructure that began at about 06:00 UTC on 10 January 2024 and broadened to more of the PHL subnet on 11 January.
- Root cause
- Component
- Contributing factor
- Detection
SourceHut's monitoring systems noticed that something was wrong and raised the alarm from about 06:00 UTC on 10 January.
- Remediation
- Got Cogent, through the colocation provider, to narrow the null route from the /24 to the targeted /32, which mostly restored service on 10 January
- Moved DNS to an external DNS provider
- Brought up a new primary database in AMS
- Restored backups from FRE to AMS
- Applied the difference between production data and the last backups by hand
- Provisioned a new mail system and configured the mail services for it
- Set up an OVH box that NATs traffic through OVH's DDoS-resistant network to the production subnet in AMS
- Used a temporary subnet from the PHL network provider for out-of-band access to the PHL servers
- Served pages.sr.ht through an out-of-band link to PHL while its object storage was migrated
- Emailed pages.sr.ht users whose apex records pointed at the PHL subnet with instructions
- Ran build workers on dedicated bare-metal OVH servers
- Action items
- Provision a new standby database
- Configure backups from AMS to FRE
- Install a new monitoring system
- Retire the PHL datacenter installation and ship its servers to AMS
- Move build service back onto SourceHut-operated servers once the build servers are shipped from PHL to AMS
- Research and implement a way to run SourceHut in an on-premise Kubernetes cluster
- Plan a second production-ready datacenter to load balance against, able to take over full service in a datacenter outage
- Lessons
The operational plans SourceHut kept for failures up to the loss of a whole datacenter proved important in this incident. The incident exposed a known shortcoming in redundancy and resilience that the team was working on but had not finished, and the response of the upstream network to the attack did more damage than anything else.
IL-0221A backbone maintenance command disconnected all Facebook data centers, and its DNS servers withdrew their BGP routesMeta2021- Symptoms
On 4 October 2021 Facebook's platforms went down worldwide: every connection in Facebook's global backbone network was lost, and the data centers were cut off from each other and from the internet. The rest of the internet could no longer find Facebook's servers, because Facebook's authoritative DNS servers had become unreachable.
- Trigger
During routine maintenance of the backbone, a command was issued to assess the availability of global backbone capacity, and it unintentionally took down all connections in the backbone network.
- Root cause
- Component
- Contributing factor
- Remediation
- Sent engineers onsite to the data centers to debug the issue and restart systems
- Brought the backbone network back online across the data center regions
- Brought services back gradually, managing the rising load to avoid crashes from a traffic surge
- Action items
- Find ways to simulate the global backbone being taken offline in storm drills
- Strengthen testing, drills and overall resilience
- Lessons
The hardening that protects Facebook's data centers against unauthorized access slowed recovery from an outage caused by Facebook's own error, and the author judges that tradeoff worth keeping. Storm drills prepared the teams to bring load back safely, although no drill had yet simulated the global backbone going offline.
IL-0222Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduledGoogle Cloud2019- Symptoms
On Sunday 2 June 2019, with user impact beginning between 11:47 and 11:49 US/Pacific, Google Cloud projects in several US regions saw elevated packet loss from network congestion, and some regions became unreachable. Customers saw higher latency, intermittent errors and lost connectivity to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1 and southamerica-east1, while us-west1, Europe and Asia had no regional congestion.
- Trigger
At 11:45 US/Pacific a relatively rare type of datacenter maintenance event started in one physical location, and the automation that runs it descheduled logical clusters that also held network control plane jobs in other physical locations.
- Root cause
- Component
- Contributing factor
- Detection
Google engineers were alerted to the failure two minutes after it began and started the incident process used for the most serious production incidents. The first customer notice appeared on the status page at 12:53 US/Pacific.
- Remediation
- Lowered the priority of less critical network traffic classes to reduce congestion
- Halted the datacenter automation software responsible for the maintenance event
- Rescheduled the network control plane and its supporting infrastructure in each location
- Rebuilt and redistributed the lost network control plane configuration, rolling it out from 14:03
- Directed service traffic away from the affected regions
- Moved the Cloud VPN control plane in three regions out of its fail-safe state
- Action items
- Keep the maintenance automation off until safeguards stop it descheduling jobs in several physical locations at once
- Harden the cluster management software to reject such requests whatever their origin
- Reconfigure the network control plane to reject maintenance requests of the type involved
- Make the network control plane in each location persist its configuration
- Let the network stay in fail static mode longer when the control plane is lost
- Review, update and test emergency response tooling and procedures, including customer communication tooling, for such network failures
- Extend continuous disaster recovery testing to this and similarly catastrophic failures
IL-0258FastMail was unreachable for almost two hours when the GRE tunnels carrying its traffic from Level 3 went downFastmail2016- Symptoms
At 4.50am Melbourne time on Sunday 7 February (evening of 6 February UTC), FastMail became unavailable and stayed unreachable for almost two hours. Traffic to FastMail's public IP range never made it into FastMail's own network.
- Trigger
The GRE tunnels between Level 3 and NYI, which carried FastMail's filtered traffic into its network, went down.
- Root cause
- Component
- Contributing factor
- Detection
NYI, the datacentre provider, noticed the failure quickly.
- Remediation
- NYI escalated with Level 3's support until FastMail's public IP ranges were pointed back to NYI
- Action items
- Go back to DDoS protection from Black Lotus under NYI's explicit control
- Build nginx frontends in the Los Angeles datacentre that reach NYI over the secondary network, and point DNS to them if NYI's primary network fails
- Add hardware in Amsterdam and change the delivery system so it depends less on NYI
- Work with NYI to understand its processes for detecting and responding to network faults
- Lessons
Much of FastMail's and NYI's reliability had come from NYI controlling its own infrastructure. With protection always on at Level 3, NYI could not restore the routes itself, and FastMail calls the resulting delay in restoring service unacceptable.
These 5 cases involve BGP routing and Backbone network (WAN). kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.
By root cause4
Configuration change1 case
- Component
- Contributing factor
- Organisation
Dependency failure2 cases
- Contributing factor
- Organisation
Operator error1 case
- Contributing factor
- Organisation
Software defect1 case
- Contributing factor
- Organisation