IL-0228 OVHcloud · 2021-10-13networking

IPv4 routing across OVHcloud's global network broke after a router change put the full internet routing table into OSPF

OVHcloud runs a global network that is fully meshed, so every router is connected to the others directly or indirectly and constantly exchanges routing information with them. After weeks of rising DDoS attacks, OVHcloud prepared a configuration change to improve its defences; the Change Advisory Board validated it per the Method of Procedures with peer review, and it was announced on 2021-10-12. The change began at 09:05 CET, and BGP isolation and configuration updates went as expected at 09:18 CET. At 09:20 CET the misinterpreted command announced the whole internet routing table into the IGP, which overloaded routers and set off the BGP and OSPF convergence loop.

The crisis management process started at 09:25 CET. After the rollback failed, a datacentre technician started work in the telecom room at 10:00 CET (3:00 am local time), first disconnecting the optical equipment to isolate connectivity. The faulty router was powered off at 10:18 CET, network convergence took two minutes, and the first services came back at 10:20 CET. With the router, named D2, offline, the network converged, the OSPF tables emptied and traffic returned to the nominal gateways; the crisis ended technically at 10:57 CET. OVHcloud's earlier update at 11:30am the same day gave 9:12am CET for the start of the intervention and 10:15am for isolating the equipment.

Severity: –
Scope: IPv4 routing across OVHcloud's global network and all OVHcloud sites; IPv6 traffic unaffected
Users affected: –
Region: global
Time to detect: 1m
Time to resolve: 1h 37m

Symptoms

At 09:20 CET on 13 October 2021, during a planned change on a router, IPv4 traffic across OVHcloud's whole network stopped being routed, while all IPv6 traffic stayed reachable. The team detected a problem with the router's performance at 09:21 CET and escalated at once.

The entire internet routing table had been announced into OVHcloud's IGP. With the OSPF table full, routers' RAM and CPU were overloaded and some routers became unstable, and a convergence loop between BGP and OSPF made IPv4 routing inoperable, so IPv4 traffic could not be processed correctly on any OVHcloud site.

The rollback procedure did not work, so at 09:30 CET the team decided to isolate the equipment physically and called for on-site support.

After the faulty router was switched off, the first services came back at 10:20 CET, and the crisis ended from a technical point of view at 10:57 CET.

Symptom Network unreachable Service outage

Root cause

Root cause Configuration change

Component Interior routing (IGP)BGP routing Backbone network (WAN)

Contributing factor Ineffective rollback Excessive blast radius

Trigger: A planned change to a router in the Vint Hill datacentre in the United States, meant to strengthen anti-DDoS protection, included a command to regulate the redistribution of BGP into OSPF, and the router did not interpret that command correctly.

Detection

At 09:21 CET the team detected a problem with the router's performance and started the escalation process.

Remediation

Disconnected the optical equipment to isolate the router's connectivity
Powered off the faulty router
Stabilised connectivity to restore the remaining services

Action items

Re-evaluate the change procedure for this type of equipment, which applies the command line natively
Strengthen the related change process

Lessons

–

Sources

https://corporate.ovhcloud.com/en/newsroom/news/network-incident/ · blog · archived copy

References

https://network.status-ovhcloud.com/incidents/rr9361xp0mh4

Related cases

IL-0225 An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet lossMicrosoft Azure · 2023
same root cause · shared BGP routing, Interior routing (IGP), Backbone network (WAN)
IL-0394 A Cloudflare router in Miami leaked IPv6 BGP routes to peers and providers for 25 minutes, congesting the backboneCloudflare · 2026
same root cause · shared BGP routing, Backbone network (WAN)
IL-0539 KT's internet service failed nationwide for about 89 minutes as wrong routes spread from one router in BusanKT · 2021
same root cause · shared BGP routing, Interior routing (IGP)
IL-0400 Public sector sites across Wales lost connectivity for over 8 hours as a PE router wrongly advertised a default routePSBA · 2026
same root cause · shared BGP routing
IL-0266 Eqiad routers looped traffic between each other after a BGP MED change and could not reach other sites' public IPsWikimedia Foundation · 2021
same root cause · shared BGP routing

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Interior routing (IGP), BGP routing, Backbone network (WAN) among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0228
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0228/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.