Cases1 case

1 to 1 of 1

1. IL-0494 Netlify's central load balancer IP stopped responding when Rackspace null-routed it in a DDoS, and A-record sites hung

   Netlify2016

   Symptoms

   At 8am PST on Monday, August 22, 2016, Netlify got alerts that its central load balancer, the single public IP that users point A records at, had stopped responding. Rackspace had null-routed the load balancer, so all traffic to that IP hung before it reached Netlify's servers.

   Network unreachable

   Trigger

   A DDoS attack against Netlify's central load balancer in Rackspace's Chicago datacenter, after which Rackspace null-routed the load balancer's public IP.

   Root cause

   Cloud provider outage

   Component

   Cloud load balancer

   Contributing factor

   Single point of failureLimited debug access

   Detection

   Alerts at 8am PST reported that the central load balancer had stopped responding, and Rackspace then confirmed it had null-routed the load balancer.

   Remediation

   • Started a new cluster of CDN machines in Google Compute behind a Google network load balancer
   • Gave clients who had contacted support the new public IP and told them to change their A records
   • Updated the documentation and announced the new IP on Twitter, Gitter and the status page

   Action items

   • Keep the new Google-based network IP running in place of the Rackspace load balancer
   • Release Netlify's own DNS hosting, integrated with the CDN, so domains do not depend on a fixed IP
   • Fix the status page publishing process so it is updated straight away during an issue
   • Build a mailer that sends targeted mails to all paying clients during an outage

   Lessons

   Netlify judged Rackspace's load balancer too vulnerable to DDoS and Rackspace's DDoS processes not up to the standard of other cloud providers, and expected Google's anycast-based Maglev load balancer to be more resilient. In the end, the only way to protect a domain from DDoS-related outages is not to expose any fixed IP address for it, which is why sites should use the www domain with a CNAME, or Netlify DNS, rather than a single A-record IP.

   View case

By root cause1