Incident Library

IL-0266 Wikimedia Foundation · 2021-11-23networking

Eqiad routers looped traffic between each other after a BGP MED change and could not reach other sites' public IPs

At about 09:37 on 23 November 2021, as part of task T295672, a change on the eqiad core routers cr1-eqiad and cr2-eqiad removed the BGP setting that sets the "MED" attribute to the OSPF cost of reaching the next hop of a BGP route. Without it, the correct routes to remote BGP destinations learnt from outside suddenly looked less preferable than the reflections of those routes that the two eqiad routers were announcing to each other. Each router chose the other as the next hop for destinations such as eqsin, and a routing loop formed between them, so eqiad could not reach hosts in other data centers over public IP addresses for about 12 minutes.

Icinga alerted at 09:39 when it could not reach the public text-lb address in eqsin. The change was reverted on both routers at 09:42, but because of how these routes update on a policy change that alone did not fix it; a forced reset of the BGP session between the eqiad routers at 09:51 did. There was no impact on end-user traffic. Task T295672 was updated with a proposal to adjust the BGP sessions so that both the "second IXP port" issue is addressed and this does not happen again.

Severity
–
Scope
eqiad could not reach hosts in other data centers, such as cache PoPs, via public IP addresses; no impact on end-user traffic
Users affected
no end-user impact
Region
eqiad
Time to detect
2m
Time to resolve
14m

Symptoms

On 23 November 2021 at 09:39, two minutes after a change on the eqiad core routers, alerts fired because Icinga in eqiad could not reach the public text-lb address in eqsin (Singapore). For about 12 minutes eqiad could not reach hosts in other data centers, such as the cache PoPs, through their public IP addresses.

Traffic for those destinations never left eqiad. After the change, cr2-eqiad, which had been sending traffic for eqsin BGP routes via codfw, preferred its local peer cr1-eqiad, and cr1-eqiad made the mirror-image choice and sent the same traffic to cr2-eqiad, so the packets bounced between the two routers in a routing loop. End-user traffic was not affected, and internal traffic mostly uses local IP subnets routed with OSPF instead of BGP; the Icinga alert was probably the only impact.

Reverting the configuration change on both routers at 09:42 did not clear the loop straight away, because of the particular way these routes update when a policy changes. After more troubleshooting, a forced reset of the BGP session between the two eqiad routers at 09:51 resolved it.

Symptom Network unreachable

Root cause

Root cause Configuration change

Component BGP routing
Contributing factor Ineffective rollback
Trigger
At about 09:37 a change on cr1-eqiad and cr2-eqiad, made as part of task T295672 to influence BGP route selection, removed a BGP setting that sets the BGP "MED" attribute to the OSPF cost of reaching the BGP route's next hop.

Detection

Icinga health checks against edge caching clusters alerted at 09:39; the first alert was Icinga in eqiad failing to reach the public text-lb address in eqsin.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, BGP routing among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.