IL-0258 Fastmail · 2016-02-06networking

FastMail was unreachable for almost two hours when the GRE tunnels carrying its traffic from Level 3 went down

After a DDoS attack in November 2015, NYI moved FastMail from its network-based DDoS protection service, Black Lotus, to a new one at Level 3, mainly for Level 3's capacity. Before the change, NYI announced FastMail's IP range (66.111.4.0/24) to its network providers with BGP and only redirected the traffic to Black Lotus, through tunnels back to NYI, when monitoring showed a large spike in traffic. After the change the protection was always on: Level 3 announced FastMail's range as part of its own network, all traffic went through Level 3, and filtered packets were sent to NYI through GRE tunnels. NYI had no direct control to change this.

At 4.50am Melbourne time on Sunday the GRE tunnels went down, so FastMail's traffic stopped reaching its network. NYI escalated with Level 3's support until FastMail's public IP ranges were pointed back to NYI, and service was restored at 6.45am (7.45pm Saturday UTC). Because Black Lotus had since added enough capacity for the attacks seen so far, FastMail went back to the old arrangement, with Black Lotus providing protection under NYI's explicit control. A secondary network between FastMail's datacentres, set up in November, kept internal systems working throughout the outage. FastMail has also moved its DNS to CloudFlare's Virtual DNS, which connects over the secondary network, so DNS keeps working. When the primary network is unavailable, FastMail's secondary incoming mail servers in Amsterdam take over, but they lack the capacity for all incoming mail and defer much of it, which delays delivery.

Severity: –
Scope: FastMail was unavailable to customers for almost two hours; internal systems kept working
Users affected: FastMail customers
Region: –
Time to detect: –
Time to resolve: 1h 55m

Symptoms

At 4.50am Melbourne time on Sunday 7 February (evening of 6 February UTC), FastMail became unavailable and stayed unreachable for almost two hours. Traffic to FastMail's public IP range never made it into FastMail's own network.

NYI, FastMail's primary datacentre provider, noticed the failure quickly but could no longer fix it itself, because all of FastMail's traffic was being routed through Level 3's DDoS protection network and only Level 3 could change that. NYI kept escalating with Level 3's support by phone until FastMail's public IP ranges were pointed back to NYI. FastMail's internal systems kept working throughout over a secondary network between its datacentres, and Pobox Mailstore customers already migrated to FastMail, who come in on a different network, were not affected.

Symptom Service outage Network unreachable

Root cause

Root cause Dependency failure

Component DDoS protection service BGP routing

Contributing factor Limited debug access

Trigger: The GRE tunnels between Level 3 and NYI, which carried FastMail's filtered traffic into its network, went down.

Detection

NYI, the datacentre provider, noticed the failure quickly.

Remediation

NYI escalated with Level 3's support until FastMail's public IP ranges were pointed back to NYI

Action items

Go back to DDoS protection from Black Lotus under NYI's explicit control
Build nginx frontends in the Los Angeles datacentre that reach NYI over the secondary network, and point DNS to them if NYI's primary network fails
Add hardware in Amsterdam and change the delivery system so it depends less on NYI
Work with NYI to understand its processes for detecting and responding to network faults

Lessons

Much of FastMail's and NYI's reliability had come from NYI controlling its own infrastructure. With protection always on at Level 3, NYI could not restore the routes itself, and FastMail calls the resulting delay in restoring service unacceptable.

Sources

https://www.fastmail.com/blog/diary-of-a-network-outage/ · blog · archived copy

Related cases

IL-0509 SourceHut was down for 170 hours after a DDoS led its upstream network to null-route the PHL datacenter's subnetSourceHut · 2024
same root cause · shared BGP routing
IL-0267 Wikimedia's eqsin cache site went unreachable when both of its transport links to the core data centers were downWikimedia Foundation · 2023
same root cause · shared BGP routing
IL-0255 Basecamp's two redundant links between data centers dropped together and a Redis replica resync loop stalled chat37signals · 2020
same root cause · shared BGP routing
IL-0358 Packet Public Cloud locations had high latency and unreachable paths from routing loops and blackholed trafficPacket · 2019
same root cause · shared BGP routing

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, DDoS protection service, BGP routing among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0258
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0258/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.