Incident Library
Applied filters Root causeConfiguration changeFactorIneffective rollbackOrganisationWikimedia FoundationClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0266 Eqiad routers looped traffic between each other after a BGP MED change and could not reach other sites' public IPs
    Wikimedia Foundation2021
    Symptoms

    On 23 November 2021 at 09:39, two minutes after a change on the eqiad core routers, alerts fired because Icinga in eqiad could not reach the public text-lb address in eqsin (Singapore). For about 12 minutes eqiad could not reach hosts in other data centers, such as the cache PoPs, through their public IP addresses.

    Network unreachable
    Trigger

    At about 09:37 a change on cr1-eqiad and cr2-eqiad, made as part of task T295672 to influence BGP route selection, removed a BGP setting that sets the BGP "MED" attribute to the OSPF cost of reaching the BGP route's next hop.

    Root cause
    Configuration change
    Component
    BGP routing
    Contributing factor
    Ineffective rollback
    Detection

    Icinga health checks against edge caching clusters alerted at 09:39; the first alert was Icinga in eqiad failing to reach the public text-lb address in eqsin.

    Remediation
    • Reverted the configuration change on both eqiad routers
    • Forced a reset of the BGP session between the two eqiad routers
    Action items
    • Updated T295672 with further detail on the incident and a proposal to adjust the BGP sessions to address the "second IXP port" issue and prevent this from happening again
    View case
Check your cluster for this

This case involves BGP routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
BGP routing
Contributing factor
Ineffective rollback
Organisation
Wikimedia Foundation
  1. IL-0266 Eqiad routers looped traffic between each other after a BGP MED change and could not reach other sites' public IPs
    Wikimedia Foundation2021