Cases1 case

1 to 1 of 1

1. IL-0590 F5 load balancers cached a path MTU of 296 after Rackspace reboots, and Mailgun connections failed intermittently

   Mailgun2014

   Symptoms

   On Saturday, September 27, 2014 at 6:50 p.m. UTC, Mailgun saw that all its database replicas and the internal traffic between its environments in all regions were broken and reporting timeouts. Once Rackspace began rebooting cloud instances in its Chicago data center (ORD), Mailgun customers had intermittent connection failures.

   Intermittent connection failuresIntermittent timeoutsPacket loss

   Trigger

   Rackspace, Mailgun's parent company and hosting provider, rebooted cloud instances in its Chicago data center (ORD), and the cloud maintenance triggered an error condition on Mailgun's F5 load balancers.

   Root cause

   DNS or network fault

   Component

   Cloud load balancerDatabase replication

   Contributing factor

   –

   Detection

   Mailgun noticed at 6:50 p.m. UTC on September 27 that all database replicas and the internal traffic between its environments were broken and reporting timeouts.

   Remediation

   • The Rackspace networking team cleared the cached path MTU on Mailgun's F5s and on some Rackspace dedicated customers' F5s
   • Set up a new virtual IP for Mailgun services
   • Changed Mailgun's DNS settings, which forced the remote F5s to clear their caches

   Action items

   • Finish the root cause analysis of the F5 path MTU problem with Rackspace
   • Make sure maintenance of Mailgun's cloud infrastructure has no impact on Mailgun customers

   View case

By root cause1