Cases2 cases

1 to 2 of 2

1. IL-0493 Netlify's Google Cloud TCP load balancer stopped serving about 5% of global traffic as all backends failed health checks

   Netlify2021

   Symptoms

   At 14:39 UTC on March 25, 2021, Netlify's monitors alerted that the health check of one of its load balancers was failing. That Google Cloud TCP load balancer served the single static IP that customers use for A records, and about 5% of Netlify's global traffic stopped being served until 18:46 UTC.

   Probe failuresHigh error rate

   Trigger

   Unattended operating system updates, whose timing Netlify randomises across production nodes, restarted the systemd-networkd unit on each node over several hours, and each restart removed the custom route table entry that held the load balancer IP.

   Root cause

   Software defect

   Component

   Cloud load balancersystemd-networkd

   Contributing factor

   Monitoring blind spotOverlapping failures

   Detection

   Netlify's monitors alerted at 14:39 UTC that the load balancer health check was failing. The load balancer's own backend health checks were not part of Netlify's alerting, so the backends dropping out over the preceding hours raised no alert.

   Remediation

   • Gave enterprise customers a separate range of IPs on a new load balancer for enterprise traffic
   • Provisioned an AWS network load balancer backed by Global Accelerator and published its IP for self-serve customers
   • Restarted the GCP networking daemon on all affected backend nodes, which restored traffic on the Google load balancer

   Action items

   • Patch the google-guest-agent systemd configuration on all GCP CDN nodes to react to systemd-networkd restarts
   • Report the bug to Google and open an issue in the google-guest-agent repository
   • Include the load balancer's backend health checks in alerting
   • Add a canary node to the backend to test updates and watch for issues over several days
   • Decouple traffic tiers and move to multiple regions
   • Offer customers more ways to route apex domains to Netlify, such as ALIAS and ANAME records
   • Route enterprise traffic through its own IP

   Lessons

   Netlify's own node-level health checks could not see that the backends were failing the load balancer's health check, and alerting on the load balancer's backend health checks would have given more time to diagnose before the outage.

   View case
2. IL-0494 Netlify's central load balancer IP stopped responding when Rackspace null-routed it in a DDoS, and A-record sites hung

   Netlify2016

   Symptoms

   At 8am PST on Monday, August 22, 2016, Netlify got alerts that its central load balancer, the single public IP that users point A records at, had stopped responding. Rackspace had null-routed the load balancer, so all traffic to that IP hung before it reached Netlify's servers.

   Network unreachable

   Trigger

   A DDoS attack against Netlify's central load balancer in Rackspace's Chicago datacenter, after which Rackspace null-routed the load balancer's public IP.

   Root cause

   Cloud provider outage

   Component

   Cloud load balancer

   Contributing factor

   Single point of failureLimited debug access

   Detection

   Alerts at 8am PST reported that the central load balancer had stopped responding, and Rackspace then confirmed it had null-routed the load balancer.

   Remediation

   • Started a new cluster of CDN machines in Google Compute behind a Google network load balancer
   • Gave clients who had contacted support the new public IP and told them to change their A records
   • Updated the documentation and announced the new IP on Twitter, Gitter and the status page

   Action items

   • Keep the new Google-based network IP running in place of the Rackspace load balancer
   • Release Netlify's own DNS hosting, integrated with the CDN, so domains do not depend on a fixed IP
   • Fix the status page publishing process so it is updated straight away during an issue
   • Build a mailer that sends targeted mails to all paying clients during an outage

   Lessons

   Netlify judged Rackspace's load balancer too vulnerable to DDoS and Rackspace's DDoS processes not up to the standard of other cloud providers, and expected Google's anycast-based Maglev load balancer to be more resilient. In the end, the only way to protect a domain from DDoS-related outages is not to expose any fixed IP address for it, which is why sites should use the www domain with a CNAME, or Netlify DNS, rather than a single A-record IP.

   View case

By root cause2