Incident Library

IL-0493 Netlify · 2021-03-25gcpnetworking

Netlify's Google Cloud TCP load balancer stopped serving about 5% of global traffic as all backends failed health checks

Google Cloud's TCP load balancer uses Direct Server Return, so each backend instance must be able to answer with the load balancer IP as its source IP. On GCP, the networking agent google-guest-agent, which ships in the GCP base image Netlify used for its backends, injects the load balancer IP into each node's local route table. On the day of the incident, an unattended update restarted systemd-networkd, which removed that route table entry. google-guest-agent watches several network manager units and re-adds the IP when they restart, but it had no watcher for systemd-networkd, so the backends were left without the IP. As the update reached each node over several hours, the node failed the load balancer's health check and stopped receiving traffic, while Netlify's own health checks, which reached the backends on their own IPs, kept reporting them healthy. Requests were dropped once systemd-networkd restarted on the last backend.

Believing the cause was the zonal L4 load balancer issue on the provider's status page, Netlify restored traffic by other routes: Google's global TCP proxy was ruled out because it does not support TCP port 80, so the team built separate load balancers for enterprise and regular traffic, gave enterprise customers alternative IPs at 15:30 UTC, and from 15:55 UTC provisioned an AWS network load balancer backed by Global Accelerator whose IP was published for self-serve customers at 16:46 UTC. At 18:38 UTC Google support pointed to a problem with the backends' virtual IP, and at 18:46 UTC restarting the GCP networking daemon on all affected nodes restored the remaining traffic on the Google load balancer.

Severity
–
Scope
Sites reached through the static IP of the Google Cloud TCP load balancer used for A records: about 5% of Netlify's global traffic was not served
Users affected
a portion of Netlify customers, those using the static A-record IP
Region
–
Time to detect
0m
Time to resolve
4h 7m

Symptoms

At 14:39 UTC on March 25, 2021, Netlify's monitors alerted that the health check of one of its load balancers was failing. That Google Cloud TCP load balancer served the single static IP that customers use for A records, and about 5% of Netlify's global traffic stopped being served until 18:46 UTC.

Netlify has two kinds of health checks for this load balancer: one connects through the load balancer IP to a backend node, and one talks to each backend directly. Throughout the incident the node-level checks reported the backends as healthy. The GCP console showed the backend nodes failing the load balancer's own health checks, but gave no further diagnostic information.

At 14:48 UTC a banner on the provider's status page read: "We are experiencing an issue with L4 load balancers in us-west1-c. Multiple managed services relying on LB and located in this zone might be affected." With nothing else to go on, the team took this as the cause and worked on mitigations, expecting a long provider resolution. The provider outage later turned out to be unrelated, and following it delayed the resolution.

Symptom Probe failuresHigh error rate

Root cause

Root cause Software defect

Component Cloud load balancersystemd-networkd
Contributing factor Monitoring blind spotOverlapping failures
Trigger
Unattended operating system updates, whose timing Netlify randomises across production nodes, restarted the systemd-networkd unit on each node over several hours, and each restart removed the custom route table entry that held the load balancer IP.

Detection

Netlify's monitors alerted at 14:39 UTC that the load balancer health check was failing. The load balancer's own backend health checks were not part of Netlify's alerting, so the backends dropping out over the preceding hours raised no alert.

Remediation

Action items

Lessons

Netlify's own node-level health checks could not see that the backends were failing the load balancer's health check, and alerting on the load balancer's backend health checks would have given more time to diagnose before the outage.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cloud load balancer, systemd-networkd among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.