Incident Library
Applied filters Root causeDependency failureSymptomNetwork unreachableClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

1 to 5 of 6

  1. IL-0526 OpenStreetMap.org went offline, then read-only, for 56 hours after HE.net routing equipment cut both fibre uplinks
    OpenStreetMap Foundation2024
    Symptoms

    At 04:01 UTC on 15 December 2024, StatusCake monitoring sent SMS alerts that OpenStreetMap.org was unreachable. The website and the mapping API had gone completely offline at 03:53 UTC.

    Network unreachableService outage
    Trigger

    HE.net's routing equipment in Amsterdam failed at 03:53 UTC on 15 December 2024.

    Root cause
    Dependency failure
    Component
    ISP and transit networksDatabase replication
    Contributing factor
    Single point of failureNo runbook
    Detection

    StatusCake monitoring detected the outage at 04:01 UTC and sent SMS alerts, and the incident was reported to the operations team at once.

    Remediation
    • Activated a read-only backup instance of OpenStreetMap.org in Dublin
    • Brought a new Equinix Internet link into service in Amsterdam, bypassing HE.net and restoring full functionality
    • HE.net replaced its routing hardware and restored its service, and routing through HE.net resumed
    Action items
    • Move to a multi-ISP architecture so another provider can take over when one fails
    • Research backup ISPs for the Amsterdam servers, including BGP and keeping HE.net as a fallback
    • Add deeper route and latency monitoring and an automated escalation protocol
    • Hold failover drills for switching between ISPs and keep replication up to date
    • Agree SLAs with HE.net and Equinix Internet with defined escalation procedures and response times
    • Prepare the 4G out-of-band system as a manual fallback uplink that can sync Postgres replication data, and upgrade its hardware
    • Document outage communication procedures and delegate communication during outages
    • Investigate decoupling the OAuth service so authentication keeps working while the database is read-only
    Lessons

    Relying on HE.net alone left the Amsterdam servers with a single point of failure, so OpenStreetMap will keep multiple upstream providers. The 4G out-of-band link proved invaluable and should be able to carry data synchronisation. Resumed replication would have restored editing faster than the read-only backup. Frequent contact with the ISP gave no useful estimates, so the SLA needs to be better understood. Alerting on upstream connectivity needs refining, and a read-only OpenStreetMap.org also blocked logins to services that were not affected, because it could not save new tokens.

    View case
  2. IL-0509 SourceHut was down for 170 hours after a DDoS led its upstream network to null-route the PHL datacenter's subnet
    SourceHut2024
    Symptoms

    At about 06:00 UTC on 10 January 2024, SourceHut's monitoring raised the alarm as a layer 3 DDoS attack hit its infrastructure in the PHL datacenter, and at about 09:00 UTC SourceHut lost all access to its PHL network. PHL was the primary datacenter, and the network became unreachable for SourceHut staff and the general public alike.

    Network unreachableService outage
    Trigger

    A layer 3 DDoS attack against SourceHut's PHL infrastructure that began at about 06:00 UTC on 10 January 2024 and broadened to more of the PHL subnet on 11 January.

    Root cause
    Dependency failure
    Component
    ISP and transit networksBGP routing
    Contributing factor
    Single point of failureLimited debug accessUntested backup
    Detection

    SourceHut's monitoring systems noticed that something was wrong and raised the alarm from about 06:00 UTC on 10 January.

    Remediation
    • Got Cogent, through the colocation provider, to narrow the null route from the /24 to the targeted /32, which mostly restored service on 10 January
    • Moved DNS to an external DNS provider
    • Brought up a new primary database in AMS
    • Restored backups from FRE to AMS
    • Applied the difference between production data and the last backups by hand
    • Provisioned a new mail system and configured the mail services for it
    • Set up an OVH box that NATs traffic through OVH's DDoS-resistant network to the production subnet in AMS
    • Used a temporary subnet from the PHL network provider for out-of-band access to the PHL servers
    • Served pages.sr.ht through an out-of-band link to PHL while its object storage was migrated
    • Emailed pages.sr.ht users whose apex records pointed at the PHL subnet with instructions
    • Ran build workers on dedicated bare-metal OVH servers
    Action items
    • Provision a new standby database
    • Configure backups from AMS to FRE
    • Install a new monitoring system
    • Retire the PHL datacenter installation and ship its servers to AMS
    • Move build service back onto SourceHut-operated servers once the build servers are shipped from PHL to AMS
    • Research and implement a way to run SourceHut in an on-premise Kubernetes cluster
    • Plan a second production-ready datacenter to load balance against, able to take over full service in a datacenter outage
    Lessons

    The operational plans SourceHut kept for failures up to the loss of a whole datacenter proved important in this incident. The incident exposed a known shortcoming in redundancy and resilience that the team was working on but had not finished, and the response of the upstream network to the attack did more damage than anything else.

    View case
  3. IL-0259 Outbound packets from Fastmail did not reach part of the internet for about 19 hours, cutting off 3-5% of customers
    Fastmail2023
    Symptoms

    From about 03:00 UTC on 30 June 2023, Fastmail and a fraction of the internet could not communicate with each other, and Fastmail first heard of it from customers through support tickets and social media. The problem lasted about 19 hours and affected an estimated 3-5% of customers as well as some email servers.

    Network unreachable
    Root cause
    Dependency failure
    Component
    ISP and transit networks
    Contributing factor
    Single point of failureMonitoring blind spot
    Detection

    Customers told Fastmail through support tickets and social media. Fastmail's own monitors in multiple locations and its staff were not on affected networks and saw nothing.

    Action items
    • Select a provider for a second transit connection directly into Fastmail's servers, via Megaport or a provider present in 365's New Jersey datacenter, so that Fastmail controls the outbound path and can route around a network with problems
    Lessons

    With a single path for outbound traffic, Fastmail could not route around a provider's network problem and could not fix it itself. Being able to choose between several outbound paths would let Fastmail get everyone back online faster.

    View case
  4. IL-0267 Wikimedia's eqsin cache site went unreachable when both of its transport links to the core data centers were down
    Wikimedia Foundation2023
    Symptoms

    On 10 January 2023 at 16:37 UTC Icinga reported "BGP status on cr3-eqsin is CRITICAL: BGP CRITICAL - AS1299/IPv6: Active - Telia, AS1299/IPv4: Active - Telia", and from 16:45 the hosts in Wikimedia's eqsin site went DOWN with "PING CRITICAL - Packet loss = 100%". Users normally directed to eqsin, mostly in the APAC region, could only read Wikipedia pages already cached in eqsin.

    Network unreachable
    Trigger

    Planned maintenance by the transport provider took down eqsin's second transport link while the first was still down from a long fiber cut.

    Root cause
    Dependency failure
    Component
    Backbone network (WAN)BGP routingCDN and edge network
    Contributing factor
    Single point of failureIgnored early signal
    Detection

    Automated monitoring: Icinga BGP status alerts for cr3-eqsin at 16:37 and cr2-eqsin at 16:44, followed by a flood of host-down alerts and three pages at the same time.

    Remediation
    • Depooled eqsin from DNS
    • Repooled eqsin
    Action items
    • Create a backup GRE tunnel for eqsin (T327265)
    • Ensure the long-down transport link comes back up properly once it is fixed (T322529)
    • Automatically parse maintenance notifications and alert on conflicting maintenance (T230835)
    Lessons

    The site was depooled quickly, and "depool first, investigate later" was the right attitude. A flood of host-down alerts usually means a network problem. eqsin stayed too long on a single working transport link, SREs did not catch that the downtime of the two transport links overlapped, and the provider's planned maintenance was not recorded in a task or reminder.

    View case
  5. IL-0358 Packet Public Cloud locations had high latency and unreachable paths from routing loops and blackholed traffic
    Packet2019
    Symptoms

    At about 08:49 UTC on 2 June 2019, Packet's internal network monitoring alerted on serious reachability and performance problems in several of Packet's global datacenter locations. The first investigation found routing loops and traffic being blackholed to specific internal and Internet destinations.

    Network unreachablePacket lossHigh latency
    Trigger

    A trans-continental US backbone link in the network of Packet's upstream provider Zayo had a hardware failure and blackholed IP traffic routed over it.

    Root cause
    Dependency failure
    Component
    Backbone network (WAN)BGP routingNetwork control plane (SDN)
    Contributing factor
    –
    Detection

    Packet's internal network monitoring alerted at 08:49 UTC on reachability issues in several global datacenter locations, and engineers confirmed them at 08:52.

    Remediation
    • Migrated traffic to an alternative network and removed Zayo from the provider mix
    • Applied route filters to return service to normal
    • Optimised specific routing policies to improve latency across the backbone with the new traffic mix
    Action items
    • Expand internal connectivity monitoring and alerting of on-call operations teams
    • Expand network automation and routing so traffic can be balanced across all available backbone paths and an operator can trigger automated re-routes across all Packet datacenters in a major provider incident
    • Make Packet's provisioning systems less dependent on third party network providers and inter-datacenter connectivity
    View case
Check your cluster for this

These 6 cases involve BGP routing and ISP and transit networks. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Dependency failure6 cases

Component
BGP routing4ISP and transit networks3Backbone network (WAN)2CDN and edge network1Database replication1DDoS protection service1Network control plane (SDN)1
Contributing factor
Single point of failure4Limited debug access2Untested backup1Ignored early signal1Monitoring blind spot1No runbook1
Organisation
Fastmail2OpenStreetMap Foundation1Packet1SourceHut1Wikimedia Foundation1
  1. IL-0526 OpenStreetMap.org went offline, then read-only, for 56 hours after HE.net routing equipment cut both fibre uplinks
    OpenStreetMap Foundation2024
  2. IL-0509 SourceHut was down for 170 hours after a DDoS led its upstream network to null-route the PHL datacenter's subnet
    SourceHut2024
  3. IL-0259 Outbound packets from Fastmail did not reach part of the internet for about 19 hours, cutting off 3-5% of customers
    Fastmail2023
  4. IL-0267 Wikimedia's eqsin cache site went unreachable when both of its transport links to the core data centers were down
    Wikimedia Foundation2023
  5. IL-0358 Packet Public Cloud locations had high latency and unreachable paths from routing loops and blackholed traffic
    Packet2019
  6. IL-0258 FastMail was unreachable for almost two hours when the GRE tunnels carrying its traffic from Level 3 went down
    Fastmail2016