Incident Library

IL-0225 Microsoft Azure · 2023-01-25azurenetworking

An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet loss

The task added new routers to Microsoft's global Wide Area Network (WAN) in Madrid: it changed each new router's IP address and joined it to the IGP (Interior Gateway Protocol, which connects all routers inside the WAN) and BGP routing domains. The standard operating procedure (SOP) for such work has four steps: testing in the Open Network Emulator, testing in the lab, a Safe-Fly Review with roll-out and roll-back plans, and Safe-Deployment, which allows access to one device at a time. The SOP had been changed before this event to fix problems from earlier runs, and the changed SOP was not re-tested and lacked proper post-checks. The added command purges the IGP database: on routers from two manufacturers it runs only on the local router, on the third it orders every IGP-joined router to recompute its topology. Microsoft's AAA system approves every router command and blocks commands with global impact, but this command's global default on that router model had not been found in the evaluation of high-impact commands and was not on the block list.

Although the engineer had access to a single router, that router was joined to the rest of the WAN through the IGP. All routers recomputed IP connectivity across the internal network, and BGP routers then readvertised and validated the prefixes received from the internet; restoring connectivity to every prefix took about 1 hour and 40 minutes. Monitoring alerted the on-call engineers within 5 minutes, but because of the unqualified SOP changes the engineer making the change was not informed and repeated the operation on the second Madrid router 33 minutes later, causing a second wave. The team reviewed recent changes and identified the command by 08:20 UTC while automatic recovery was under way. Nearly all network devices and most regions and services had recovered by 09:05 UTC, and the last network equipment by 09:25 UTC. During the event the automated systems that keep the WAN healthy, including the one that removes unhealthy devices and the traffic engineering system, had been paused, so some paths stayed unoptimised and lost packets from 09:25 UTC until those systems were restarted by hand; recovery completed at 12:43 UTC.

Severity
–
Scope
Network connectivity to Azure regions and to other Microsoft services including Microsoft 365 and Power Platform, internet, inter-region and ExpressRoute or VPN/vWAN traffic, and Azure Government services that depend on the public cloud
Users affected
–
Region
global
Time to detect
3m
Time to resolve
5h 35m

Symptoms

From 07:08 UTC on 25 January 2023, customers saw long network latency and timeouts when connecting to resources in Azure regions and to other Microsoft services such as Microsoft 365 and Power Platform. Microsoft's monitoring detected DNS and WAN issues from 07:11 UTC.

The impact changed as routing converged across the network: some customers had intermittent connectivity, some saw connections time out, and others had long latency or lost connectivity completely. It hit traffic to and from the internet, traffic between regions, cross-premises traffic over ExpressRoute or VPN/vWAN, and Azure Government services that depend on the public cloud.

Alerts reached the on-call engineers within 5 minutes of the command, but the engineer making the change was not told. The same operation was run on a second router in Madrid 33 minutes after the first, which sent a second wave of connectivity issues through the network. Because of the size of the network, restoring connectivity to every prefix took about 1 hour and 40 minutes. Responders began by reviewing all recent changes and identified the command at 08:20 UTC while automatic recovery was under way.

Most regions and services recovered by 09:05 UTC and the last network equipment by 09:25 UTC, but parts of the network kept above-normal packet loss, and two routers could not recover automatically. Traffic crossing the regions in India and parts of North America recovered last, at 12:43 UTC.

Symptom High latencyIntermittent timeoutsIntermittent connection failuresPacket lossNetwork unreachable

Root cause

Root cause Configuration change

Component Backbone network (WAN)Interior routing (IGP)BGP routing
Contributing factor No configuration testsUnverified assumptionExcessive blast radiusHarmful manual action
Trigger
A network engineer adding capacity to the global WAN in Madrid ran a step of a recently changed standard operating procedure that purged the IGP database; on the router manufacturer involved, that command runs on all IGP-joined routers instead of only the local one.

Detection

Monitoring detected DNS and WAN issues from 07:11 UTC and alerted the on-call engineers within 5 minutes of the command being run. The team reviewed all recent changes and identified the command by 08:20 UTC. The engineer making the change was not informed of the alerts because of the unqualified changes to the SOP.

Remediation

Action items

Lessons

–

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Backbone network (WAN), Interior routing (IGP), BGP routing among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.