Incident Library

Organisation

Netlify

Netlify · netlify

2 cases · 2016 to 2021

Root cause Cloud provider outage1Software defect1
Component Cloud load balancer2systemd-networkd1
Contributing factor Limited debug access1Monitoring blind spot1Overlapping failures1Single point of failure1
Symptom High error rate1Network unreachable1Probe failures1

Cases

  1. IL-0493 Netlify's Google Cloud TCP load balancer stopped serving about 5% of global traffic as all backends failed health checksNetlify · 2021

    Root cause Software defect

  2. IL-0494 Netlify's central load balancer IP stopped responding when Rackspace null-routed it in a DDoS, and A-record sites hungNetlify · 2016

    Root cause Cloud provider outage