Incident Library

IL-0259 Fastmail · 2023-06-30networking

Outbound packets from Fastmail did not reach part of the internet for about 19 hours, cutting off 3-5% of customers

Fastmail's servers are in a datacenter in Bridgewater, New Jersey that belonged to New York Internet (NYI) and was later sold to 365 Datacenters. Fastmail still uses the IP range 66.111.4.0/24, which belongs to NYI and is leased to Fastmail because IP reputation matters for email; this has made routing more complex, and Fastmail has spent the past year moving to a new range while running both. Incoming traffic arrives through Cloudflare's MagicTransit, which routes at the IP layer for DDoS protection, either over Fastmail's network IP on the 365 managed network or, by preference, over a fiber link to Megaport.

Outgoing traffic had only one path to the internet. On 30 June 2023, from about 03:00 to about 22:00 UTC, packets sent out over that path did not reach a fraction of the internet, while incoming traffic over Megaport kept working. With a single outbound path, Fastmail could not route around the problem itself; it describes the failure as outside its control but its responsibility, and says it is quite certain the root cause is now fixed. Fastmail plans a second transit connection directly into its servers so that it can choose the path for outbound traffic and route around a network with problems.

Severity
–
Scope
All Fastmail services, including mail delivery, between Fastmail and the affected networks: Fastmail entirely offline for users on them, delayed email, some mail fetches disabled, and no outgoing email to part of the internet
Users affected
An estimated 3-5% of Fastmail customers, and some email servers
Region
A fraction of the internet
Time to detect
–
Time to resolve
19h

Symptoms

From about 03:00 UTC on 30 June 2023, Fastmail and a fraction of the internet could not communicate with each other, and Fastmail first heard of it from customers through support tickets and social media. The problem lasted about 19 hours and affected an estimated 3-5% of customers as well as some email servers.

Fastmail's monitors in several locations around the world, and its staff, were not on any of the affected networks, so Fastmail could not see the failing links itself. The reports showed how isolated it was: some users could connect from their phone but not from their home network, or the other way round, and some found a working path through a VPN. The outage delayed email, disabled some mail fetches and kept Fastmail from sending email to part of the internet, and for users on the affected networks Fastmail was entirely offline.

Incoming traffic kept arriving reliably over Megaport during the whole incident; it was packets leaving Fastmail's network that did not all reach their destination. Customers on affected networks sent traceroutes, which gave Fastmail addresses to trace back to from its own side and see where the traffic was dropped.

Symptom Network unreachable

Root cause

Root cause Dependency failure

Component ISP and transit networks
Contributing factor Single point of failureMonitoring blind spot
Trigger
–

Detection

Customers told Fastmail through support tickets and social media. Fastmail's own monitors in multiple locations and its staff were not on affected networks and saw nothing.

Remediation

–

Action items

Lessons

With a single path for outbound traffic, Fastmail could not route around a provider's network problem and could not fix it itself. Being able to choose between several outbound paths would let Fastmail get everyone back online faster.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, ISP and transit networks among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.