Incident Library
Applied filters Root causeDependency failureSymptomNetwork unreachableClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

6 to 6 of 6

  1. IL-0258 FastMail was unreachable for almost two hours when the GRE tunnels carrying its traffic from Level 3 went down
    Fastmail2016
    Symptoms

    At 4.50am Melbourne time on Sunday 7 February (evening of 6 February UTC), FastMail became unavailable and stayed unreachable for almost two hours. Traffic to FastMail's public IP range never made it into FastMail's own network.

    Service outageNetwork unreachable
    Trigger

    The GRE tunnels between Level 3 and NYI, which carried FastMail's filtered traffic into its network, went down.

    Root cause
    Dependency failure
    Component
    DDoS protection serviceBGP routing
    Contributing factor
    Limited debug access
    Detection

    NYI, the datacentre provider, noticed the failure quickly.

    Remediation
    • NYI escalated with Level 3's support until FastMail's public IP ranges were pointed back to NYI
    Action items
    • Go back to DDoS protection from Black Lotus under NYI's explicit control
    • Build nginx frontends in the Los Angeles datacentre that reach NYI over the secondary network, and point DNS to them if NYI's primary network fails
    • Add hardware in Amsterdam and change the delivery system so it depends less on NYI
    • Work with NYI to understand its processes for detecting and responding to network faults
    Lessons

    Much of FastMail's and NYI's reliability had come from NYI controlling its own infrastructure. With protection always on at Level 3, NYI could not restore the routes itself, and FastMail calls the resulting delay in restoring service unacceptable.

    View case
Check your cluster for this

These 6 cases involve BGP routing and ISP and transit networks. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Dependency failure6 cases

Component
BGP routing4ISP and transit networks3Backbone network (WAN)2CDN and edge network1Database replication1DDoS protection service1Network control plane (SDN)1
Contributing factor
Single point of failure4Limited debug access2Untested backup1Ignored early signal1Monitoring blind spot1No runbook1
Organisation
Fastmail2OpenStreetMap Foundation1Packet1SourceHut1Wikimedia Foundation1
  1. IL-0526 OpenStreetMap.org went offline, then read-only, for 56 hours after HE.net routing equipment cut both fibre uplinks
    OpenStreetMap Foundation2024
  2. IL-0509 SourceHut was down for 170 hours after a DDoS led its upstream network to null-route the PHL datacenter's subnet
    SourceHut2024
  3. IL-0259 Outbound packets from Fastmail did not reach part of the internet for about 19 hours, cutting off 3-5% of customers
    Fastmail2023
  4. IL-0267 Wikimedia's eqsin cache site went unreachable when both of its transport links to the core data centers were down
    Wikimedia Foundation2023
  5. IL-0358 Packet Public Cloud locations had high latency and unreachable paths from routing loops and blackholed traffic
    Packet2019
  6. IL-0258 FastMail was unreachable for almost two hours when the GRE tunnels carrying its traffic from Level 3 went down
    Fastmail2016