Incident Library

IL-0494 Netlify · 2016-08-22networkingsecurity

Netlify's central load balancer IP stopped responding when Rackspace null-routed it in a DDoS, and A-record sites hung

Netlify served sites in two ways: through the Netlify CDN, with points of presence worldwide and traffic routed to the closest location, and through a central load balancer whose IP users can target with A records when their DNS provider cannot set a CNAME on a naked domain. Until August 22, 2016 that load balancer was provisioned in Rackspace's Chicago datacenter. When DDoS traffic overwhelmed it, Rackspace null-routed it, and handling the attack was out of Netlify's hands because the load balancer was a managed one.

With no timeline from Rackspace, Netlify started a new cluster of CDN machines in Google Compute behind a Google network load balancer based on Maglev. Clients who had contacted support were given the new public IP and told to change their A records to bring their naked domains back online, and once the setup proved robust, Netlify updated its documentation and announced the new IP on Twitter, Gitter and its status page. Updating the status page took longer than it should have, because its separate publishing system, built on an external CI provider, S3 and Cloudfront, turned out to be fragile.

Severity
–
Scope
All sites with an A record pointed at Netlify's central load balancer IP; naked domains used as primary domains were hit hardest
Users affected
Netlify users with A records pointed at the load balancer IP
Region
–
Time to detect
–
Time to resolve
–

Symptoms

At 8am PST on Monday, August 22, 2016, Netlify got alerts that its central load balancer, the single public IP that users point A records at, had stopped responding. Rackspace had null-routed the load balancer, so all traffic to that IP hung before it reached Netlify's servers.

Every site with an A record pointed at that IP was affected to some degree, and sites that used a naked domain with an A record as their primary domain were hit the hardest. Sites served through the Netlify CDN with a CNAME did not go through this load balancer.

Two days earlier, on the night of Saturday, August 20, the load balancer had shown a sharp surge in incoming transfer with no matching rise in outgoing requests. Netlify added capacity behind the load balancer and asked Rackspace whether the traffic was legitimate or the start of a DDoS attack; Rackspace said the level of traffic was nothing to worry about. During the outage, even after escalating to the account manager, Netlify could get no ETA and no data on the size or type of the attack.

Symptom Network unreachable

Root cause

Root cause Cloud provider outage

Component Cloud load balancer
Contributing factor Single point of failureLimited debug access
Trigger
A DDoS attack against Netlify's central load balancer in Rackspace's Chicago datacenter, after which Rackspace null-routed the load balancer's public IP.

Detection

Alerts at 8am PST reported that the central load balancer had stopped responding, and Rackspace then confirmed it had null-routed the load balancer.

Remediation

Action items

Lessons

Netlify judged Rackspace's load balancer too vulnerable to DDoS and Rackspace's DDoS processes not up to the standard of other cloud providers, and expected Google's anycast-based Maglev load balancer to be more resilient. In the end, the only way to protect a domain from DDoS-related outages is not to expose any fixed IP address for it, which is why sites should use the www domain with a CNAME, or Netlify DNS, rather than a single A-record IP.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cloud load balancer among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.