IL-0590 Mailgun · 2014-09-27networking

F5 load balancers cached a path MTU of 296 after Rackspace reboots, and Mailgun connections failed intermittently

Mailgun runs several environments in multiple Rackspace data centers on a hybrid cloud setup in which F5 load balancers are the primary gateway for all incoming and outgoing connections. During outages or maintenance Mailgun moves traffic between environments by changing the load balancing ratios of F5 virtual IP pools, without DNS changes. On Friday, September 26, Rackspace told Mailgun it would reboot cloud instances in ORD between Sunday, September 28, 11:00 a.m. UTC and Monday, September 29, 11:00 a.m. UTC. Mailgun prepared to switch traffic away from ORD, but on September 27 its database replicas and the traffic between environments started timing out, and Mailgun could not find why before the reboots began.

The cloud maintenance put Mailgun's F5 servers into an error condition in which they reported a path MTU of 296 for the IPs on Mailgun's networks, and the F5s of all Rackspace dedicated customers using Mailgun cached that value. The servers behind those F5s could only send packets with a minimum MTU of 512, while Mailgun's F5 enforced the smaller value, so packets were lost. The Rackspace networking team cleared the caches on Mailgun's F5s and on some dedicated customers' F5s and set up a new virtual IP for Mailgun services, and Mailgun changed DNS settings, which forced the remote F5s to clear their caches. Mailgun said all messages reported as accepted during the outage were delivered. When the post was written, Mailgun and Rackspace were still investigating the root cause of the unusually low path MTU value.

Severity: –
Scope: Connections to Mailgun: intermittent connection failures, events lost between 2:30 a.m. and 7:00 a.m. UTC, and approximately 100,000 duplicate emails sent
Users affected: A portion of Mailgun customers, including Rackspace dedicated customers using Mailgun
Region: Rackspace Chicago data center (ORD); internal traffic between Mailgun's environments in all regions
Time to detect: –
Time to resolve: –

Symptoms

On Saturday, September 27, 2014 at 6:50 p.m. UTC, Mailgun saw that all its database replicas and the internal traffic between its environments in all regions were broken and reporting timeouts. Once Rackspace began rebooting cloud instances in its Chicago data center (ORD), Mailgun customers had intermittent connection failures.

The reboots began at 1:10 a.m. UTC on September 30, and from then until 11:29 a.m. UTC customers saw intermittent connection failures, events were lost between 2:30 a.m. and 7:00 a.m. UTC, and approximately 100,000 duplicate emails were sent. A portion of Mailgun customers kept having intermittent connection loss and failures through the afternoon of Tuesday, September 30.

Mailgun had prepared to move traffic away from the ORD environment before the reboots, but it could not find the cause of the timeouts before the reboots started, so traffic was not moved in time. The source of the problem was found later, together with the Rackspace enterprise networking team.

Symptom Intermittent connection failures Intermittent timeouts Packet loss

Root cause

Root cause DNS or network fault

Component Cloud load balancer Database replication

Contributing factor –

Trigger: Rackspace, Mailgun's parent company and hosting provider, rebooted cloud instances in its Chicago data center (ORD), and the cloud maintenance triggered an error condition on Mailgun's F5 load balancers.

Detection

Mailgun noticed at 6:50 p.m. UTC on September 27 that all database replicas and the internal traffic between its environments were broken and reporting timeouts.

Remediation

The Rackspace networking team cleared the cached path MTU on Mailgun's F5s and on some Rackspace dedicated customers' F5s
Set up a new virtual IP for Mailgun services
Changed Mailgun's DNS settings, which forced the remote F5s to clear their caches

Action items

Finish the root cause analysis of the F5 path MTU problem with Rackspace
Make sure maintenance of Mailgun's cloud infrastructure has no impact on Mailgun customers

Lessons

–

Sources

https://www.mailgun.com/blog/mailgun-post-mortem-september-2014/ · blog · archived copy

References

http://www.rackspace.com/blog/an-apology/

Related cases

IL-0583 Service-to-service calls through Amazon VPC Lattice failed a few times a day with read ECONNRESETUniform Next · 2025
same root cause · shared Cloud load balancer
IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALBLotte E-commerce · 2022
same root cause · shared Cloud load balancer
IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicasCloudflare · 2020
same root cause · shared Database replication
IL-0210 GitHub's MySQL clusters failed over to the US West Coast and the site served slow, stale, inconsistent data for 24 hoursGitHub · 2018
same root cause · shared Database replication
IL-0351 Heroku apps in the EU and US had slow and failed requests as DNS round-robin sent traffic to single router hostsHeroku · 2017
same root cause · shared Cloud load balancer

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cloud load balancer, Database replication among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0590
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0590/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.