Incident Library

IL-0267 Wikimedia Foundation · 2023-01-10networking

Wikimedia's eqsin cache site went unreachable when both of its transport links to the core data centers were down

Wikimedia's eqsin site reaches the core data centers over two transport links. One of them had been out for a long time because of a fiber cut, so eqsin was running on a single link. The transport provider's planned work PWIC225900, announced by Arelion on 22 December 2022 with a reminder on 9 January, started its service window at 16:00 UTC on 10 January and took down the remaining link. The maintenance was not tracked in a task or reminder, and SREs did not catch that the two link outages overlapped.

At 16:37 and 16:44 Icinga reported the BGP sessions on cr3-eqsin and cr2-eqsin as CRITICAL, and from 16:45 the eqsin hosts, routers and load balancer addresses went DOWN with 100% packet loss, setting off a flood of alerts and three simultaneous pages. At 16:48 eqsin was depooled from DNS, following an existing runbook, so users were sent elsewhere once their DNS resolvers picked up the change; until then they could only read pages already cached in eqsin. The hosts came back from 16:55, and eqsin was later repooled. Users in Asia were affected for about 11 to 41 minutes; the incident record runs from 16:00 to 20:57.

Severity
–
Scope
Users normally directed to eqsin could read only Wikipedia pages already cached in eqsin
Users affected
users in Asia, mostly in the APAC region, for about 11 to 41 minutes
Region
APAC (eqsin)
Time to detect
–
Time to resolve
4h 57m

Symptoms

On 10 January 2023 at 16:37 UTC Icinga reported "BGP status on cr3-eqsin is CRITICAL: BGP CRITICAL - AS1299/IPv6: Active - Telia, AS1299/IPv4: Active - Telia", and from 16:45 the hosts in Wikimedia's eqsin site went DOWN with "PING CRITICAL - Packet loss = 100%". Users normally directed to eqsin, mostly in the APAC region, could only read Wikipedia pages already cached in eqsin.

At 16:44 a second BGP alert followed: "BGP status on cr2-eqsin is CRITICAL: BGP CRITICAL - No response from remote host 103.102.166.130". Between 16:45 and 16:47 host-down alerts arrived for bast5002, doh5001, prometheus5001, ncredir5001, netflow5002, cr2-eqsin, durum5002, cr3-eqsin, ncredir5002, durum5001, install5001, doh5002, upload-lb.eqsin.wikimedia.org and text-lb.eqsin.wikimedia.org, and from 16:49 to 16:50 for the IPv6 checks of cr2-eqsin, mr1-eqsin, cr3-eqsin and ripe-atlas-eqsin. The alert volume was hard to handle on IRC: three pages fired at the same time and two of them escalated to batphone.

eqsin is connected to the core data centers by two transport links. One had been down for a long time from a fiber cut (T322529), and the other went down in a planned maintenance by the transport provider. Arelion had announced planned work PWIC225900 on 22 December and sent a reminder on 9 January, and its service window started at 16:00, but the maintenance was not in a task or reminder and SREs did not notice that it overlapped with the other link's outage.

Hosts recovered with "PING OK - Packet loss = 0%" from 16:55, and "OSPF status on cr1-codfw is OK" followed at 16:57. The user impact lasted about 11 minutes plus the time users' DNS resolvers took to pick up the depool of eqsin, with a long tail of up to 30 minutes.

Symptom Network unreachable

Root cause

Root cause Dependency failure

Component Backbone network (WAN)BGP routingCDN and edge network
Contributing factor Single point of failureIgnored early signal
Trigger
Planned maintenance by the transport provider took down eqsin's second transport link while the first was still down from a long fiber cut.

Detection

Automated monitoring: Icinga BGP status alerts for cr3-eqsin at 16:37 and cr2-eqsin at 16:44, followed by a flood of host-down alerts and three pages at the same time.

Remediation

Action items

Lessons

The site was depooled quickly, and "depool first, investigate later" was the right attitude. A flood of host-down alerts usually means a network problem. eqsin stayed too long on a single working transport link, SREs did not catch that the downtime of the two transport links overlapped, and the provider's planned maintenance was not recorded in a task or reminder.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Backbone network (WAN), BGP routing, CDN and edge network among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.