Incident Library
Applied filters ComponentCloud load balancerSymptomLoad imbalanceClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0352 Heroku Router 2.0 spread HTTP requests unevenly across dynos for four days, causing latency and timeouts
    Heroku2025
    Symptoms

    From about 22:00 UTC on 18 July 2025, some apps on Heroku's Common Runtime that used Router 2.0 had their HTTP requests balanced unevenly across their web dynos. The dynos that got more requests were overloaded, and the apps saw higher HTTP request latency, timeouts, reduced availability and various Heroku error codes in application logs and dashboards.

    Load imbalanceHigh latencyIntermittent timeouts
    Trigger

    From 22:00 UTC on 18 July 2025 router instances began hitting a code condition that broke the entropy metadata Router 2.0 uses to choose dynos. The condition had been in the code since the Router 2.0 beta and became more likely as more HTTP requests went through the new router, with about 94% of Common Runtime apps migrated.

    Root cause
    Software defect
    Component
    Cloud load balancer
    Contributing factor
    Monitoring blind spotUnrepresentative staging
    Detection

    No alert fired. Heroku Support received the first customer reports of uneven request balancing at 16:46 UTC on 20 July 2025, and an internal investigation was declared on 22 July.

    Remediation
    • Advised affected customers to disable Router 2.0 and fall back to the legacy router
    • Cycled the production router fleets, which improved load balancing for affected apps
    • Deployed the router fix to staging, then rolled it out to all Common Runtime partitions by 00:12 UTC on 23 July
    Action items
    • Introduced gating and notifications so that load balancing behaviour regressions do not occur
    • Communicate the timeline and send reminders to apps still on the legacy router before automated migrations resume
    View case
  2. IL-0543 Twingate API traffic piled onto one backend behind the global load balancer, failing core services for 29 minutes
    Twingate2025
    Symptoms

    At 20:45 UTC on 28 May 2025, most of Twingate's API traffic started going to a single backend service, which was overwhelmed and returned elevated error rates, so API calls failed widely across core services.

    Misrouted requestsLoad imbalanceHigh error rate
    Trigger

    A rollout that updated the routing rules of Twingate's global load balancer.

    Root cause
    Configuration change
    Component
    Cloud load balancer
    Contributing factor
    Unrepresentative stagingCircular dependency
    Detection

    The incident was detected at 20:48 UTC, three minutes after API traffic was first misrouted, and triage began.

    Remediation
    • Started a rollback of the routing change at 20:52 UTC; the automated rollback stalled on infrastructure dependencies
    • Applied a manual fix with local tooling at 21:09 UTC, which restored service by 21:14 UTC
    Action items
    • Correct the routing rule
    • Add alerts that detect backend traffic anomalies in lower environments
    • Improve resource scaling thresholds
    • Decouple infrastructure tooling from service dependencies so recovery has faster options
    • Introduce backend sharding to isolate customer workloads and reduce the impact of incidents
    View case
  3. IL-0351 Heroku apps in the EU and US had slow and failed requests as DNS round-robin sent traffic to single router hosts
    Heroku2017
    Symptoms

    Between 3 and 7 June 2017, Heroku customers in both the EU and US regions had periods of high latency: some requests failed and others took an excessive time to pass through the Heroku Routing Layer. All customers in the EU and US were affected at various times.

    High latencyHigh error rateLoad imbalance
    Root cause
    DNS or network fault
    Component
    Authoritative DNS servers and recordsCloud load balancer
    Contributing factor
    Unverified assumption
    Remediation
    • Found the sources of high-volume traffic and worked with those apps' owners to distribute their requests better
    • Replaced the legacy DNS records with new, corrected DNS records
    Action items
    • Make monitoring alert engineers sooner to unevenly distributed traffic
    • Investigate how to configure DNS and load balancers to reduce the effect of high-volume and bursting traffic
    • Work with caching Add-on partners on the DNS configuration issues that affect them, to achieve proper load balancing
    View case
Check your cluster for this

These 3 cases involve Cloud load balancer and Authoritative DNS servers and records. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause3

Software defect1 case

Component
Cloud load balancer
Contributing factor
Monitoring blind spotUnrepresentative staging
Organisation
Heroku
  1. IL-0352 Heroku Router 2.0 spread HTTP requests unevenly across dynos for four days, causing latency and timeouts
    Heroku2025

Configuration change1 case

Component
Cloud load balancer
Contributing factor
Circular dependencyUnrepresentative staging
Organisation
Twingate
  1. IL-0543 Twingate API traffic piled onto one backend behind the global load balancer, failing core services for 29 minutes
    Twingate2025

DNS or network fault1 case

Component
Cloud load balancerAuthoritative DNS servers and records
Contributing factor
Unverified assumption
Organisation
Heroku
  1. IL-0351 Heroku apps in the EU and US had slow and failed requests as DNS round-robin sent traffic to single router hosts
    Heroku2017