Cases1 case

1 to 1 of 1

1. IL-0493 Netlify's Google Cloud TCP load balancer stopped serving about 5% of global traffic as all backends failed health checks

   Netlify2021

   Symptoms

   At 14:39 UTC on March 25, 2021, Netlify's monitors alerted that the health check of one of its load balancers was failing. That Google Cloud TCP load balancer served the single static IP that customers use for A records, and about 5% of Netlify's global traffic stopped being served until 18:46 UTC.

   Probe failuresHigh error rate

   Trigger

   Unattended operating system updates, whose timing Netlify randomises across production nodes, restarted the systemd-networkd unit on each node over several hours, and each restart removed the custom route table entry that held the load balancer IP.

   Root cause

   Software defect

   Component

   Cloud load balancersystemd-networkd

   Contributing factor

   Monitoring blind spotOverlapping failures

   Detection

   Netlify's monitors alerted at 14:39 UTC that the load balancer health check was failing. The load balancer's own backend health checks were not part of Netlify's alerting, so the backends dropping out over the preceding hours raised no alert.

   Remediation

   • Gave enterprise customers a separate range of IPs on a new load balancer for enterprise traffic
   • Provisioned an AWS network load balancer backed by Global Accelerator and published its IP for self-serve customers
   • Restarted the GCP networking daemon on all affected backend nodes, which restored traffic on the Google load balancer

   Action items

   • Patch the google-guest-agent systemd configuration on all GCP CDN nodes to react to systemd-networkd restarts
   • Report the bug to Google and open an issue in the google-guest-agent repository
   • Include the load balancer's backend health checks in alerting
   • Add a canary node to the backend to test updates and watch for issues over several days
   • Decouple traffic tiers and move to multiple regions
   • Offer customers more ways to route apex domains to Netlify, such as ALIAS and ANAME records
   • Route enterprise traffic through its own IP

   Lessons

   Netlify's own node-level health checks could not see that the backends were failing the load balancer's health check, and alerting on the load balancer's backend health checks would have given more time to diagnose before the outage.

   View case

By root cause1