IL-0351 Heroku · 2017-06-03networking
Heroku apps in the EU and US had slow and failed requests as DNS round-robin sent traffic to single router hosts
The DNS records for your-app.herokuapp.com are meant to spread traffic over Heroku's routing fleet: each lookup should return a random subset of router addresses. Heroku expected its DNS provider to return several addresses, but for Heroku's specific use case the provider returned only one, a behaviour the provider calls legacy. For traffic patterns such as many HTTP clients sharing one caching DNS resolver, a large number of requests therefore went to a single router host, which was overwhelmed, and requests became slow or failed. During the latency periods Heroku engineers found the sources of high-volume traffic and worked with the owners of those apps to distribute their requests better. The provider has since changed its systems to return multiple addresses for new records, and Heroku replaced its legacy DNS records with new ones, after which incoming traffic was spread more evenly. While investigating, Heroku also found DNS configuration issues that affect its caching Add-on providers.
- Severity
- –
- Scope
- Requests through the Heroku Routing Layer were slow or failed during periods between 3 and 7 June 2017
- Users affected
- All customers in the EU and US, at various times
- Region
- EU and US
- Time to detect
- –
- Time to resolve
- –
Symptoms
Between 3 and 7 June 2017, Heroku customers in both the EU and US regions had periods of high latency: some requests failed and others took an excessive time to pass through the Heroku Routing Layer. All customers in the EU and US were affected at various times.
The slowness came from hot-spotting. A disproportionate share of traffic reached individual servers in the Routing Layer and overwhelmed them, for example when a large group of HTTP clients used a single caching DNS resolver and a large number of requests went to a single host.
The status page entry itself records one window on 7 June: from 20:50 UTC (16:50 EDT), apps in the US Common Runtime using non-SSL/SNI endpoints had increased routing latency, with the same impact as Heroku incident 1180, and the entry was resolved at 21:50 UTC.
Root cause
Root cause DNS or network fault
- Trigger
- –
Detection
–
Remediation
- Found the sources of high-volume traffic and worked with those apps' owners to distribute their requests better
- Replaced the legacy DNS records with new, corrected DNS records
Action items
- Make monitoring alert engineers sooner to unevenly distributed traffic
- Investigate how to configure DNS and load balancers to reduce the effect of high-volume and bursting traffic
- Work with caching Add-on partners on the DNS configuration issues that affect them, to achieve proper load balancing
Lessons
–
Sources
- https://status.heroku.com/incidents/1182 · official-status · archived copy
References
Related cases
IL-0583Service-to-service calls through Amazon VPC Lattice failed a few times a day with read ECONNRESETUniform Next · 2025same root cause · shared Cloud load balancer
IL-0178On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALBLotte E-commerce · 2022same root cause · shared Cloud load balancer
IL-0590F5 load balancers cached a path MTU of 296 after Rackspace reboots, and Mailgun connections failed intermittentlyMailgun · 2014same root cause · shared Cloud load balancer
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0351
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0351/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.