Cases2 cases

1 to 2 of 2

1. IL-0352 Heroku Router 2.0 spread HTTP requests unevenly across dynos for four days, causing latency and timeouts

   Heroku2025

   Symptoms

   From about 22:00 UTC on 18 July 2025, some apps on Heroku's Common Runtime that used Router 2.0 had their HTTP requests balanced unevenly across their web dynos. The dynos that got more requests were overloaded, and the apps saw higher HTTP request latency, timeouts, reduced availability and various Heroku error codes in application logs and dashboards.

   Load imbalanceHigh latencyIntermittent timeouts

   Trigger

   From 22:00 UTC on 18 July 2025 router instances began hitting a code condition that broke the entropy metadata Router 2.0 uses to choose dynos. The condition had been in the code since the Router 2.0 beta and became more likely as more HTTP requests went through the new router, with about 94% of Common Runtime apps migrated.

   Root cause

   Software defect

   Component

   Cloud load balancer

   Contributing factor

   Monitoring blind spotUnrepresentative staging

   Detection

   No alert fired. Heroku Support received the first customer reports of uneven request balancing at 16:46 UTC on 20 July 2025, and an internal investigation was declared on 22 July.

   Remediation

   • Advised affected customers to disable Router 2.0 and fall back to the legacy router
   • Cycled the production router fleets, which improved load balancing for affected apps
   • Deployed the router fix to staging, then rolled it out to all Common Runtime partitions by 00:12 UTC on 23 July

   Action items

   • Introduced gating and notifications so that load balancing behaviour regressions do not occur
   • Communicate the timeline and send reminders to apps still on the legacy router before automated migrations resume

   View case
2. IL-0493 Netlify's Google Cloud TCP load balancer stopped serving about 5% of global traffic as all backends failed health checks

   Netlify2021

   Symptoms

   At 14:39 UTC on March 25, 2021, Netlify's monitors alerted that the health check of one of its load balancers was failing. That Google Cloud TCP load balancer served the single static IP that customers use for A records, and about 5% of Netlify's global traffic stopped being served until 18:46 UTC.

   Probe failuresHigh error rate

   Trigger

   Unattended operating system updates, whose timing Netlify randomises across production nodes, restarted the systemd-networkd unit on each node over several hours, and each restart removed the custom route table entry that held the load balancer IP.

   Root cause

   Software defect

   Component

   Cloud load balancersystemd-networkd

   Contributing factor

   Monitoring blind spotOverlapping failures

   Detection

   Netlify's monitors alerted at 14:39 UTC that the load balancer health check was failing. The load balancer's own backend health checks were not part of Netlify's alerting, so the backends dropping out over the preceding hours raised no alert.

   Remediation

   • Gave enterprise customers a separate range of IPs on a new load balancer for enterprise traffic
   • Provisioned an AWS network load balancer backed by Global Accelerator and published its IP for self-serve customers
   • Restarted the GCP networking daemon on all affected backend nodes, which restored traffic on the Google load balancer

   Action items

   • Patch the google-guest-agent systemd configuration on all GCP CDN nodes to react to systemd-networkd restarts
   • Report the bug to Google and open an issue in the google-guest-agent repository
   • Include the load balancer's backend health checks in alerting
   • Add a canary node to the backend to test updates and watch for issues over several days
   • Decouple traffic tiers and move to multiple regions
   • Offer customers more ways to route apex domains to Netlify, such as ALIAS and ANAME records
   • Route enterprise traffic through its own IP

   Lessons

   Netlify's own node-level health checks could not see that the backends were failing the load balancer's health check, and alerting on the load balancer's backend health checks would have given more time to diagnose before the outage.

   View case

By root cause1