Cases2 cases

1 to 2 of 2

1. IL-0262 Requests through one GitHub datacenter failed or slowed after its load balancer nodes ran out of network flows

   GitHub2026

   Symptoms

   On 17 August 2026 from 13:40 UTC, GitHub issues and pull requests returned errors and slow responses on their pages and in the API. Automated monitoring detected the raised errors, and many other services that route through the same datacenter's load balancers, including the REST and GraphQL APIs, Actions, Copilot, sign-in and authentication, and the webhooks API, also showed errors and latency.

   High error rateHigh latency

   Trigger

   A new peak in traffic pushed the load balancers of one GitHub datacenter past their limits.

   Root cause

   Capacity exhaustion

   Component

   On-premises load balancerService mesh

   Contributing factor

   Retry storm

   Detection

   Automated monitoring detected the elevated errors and an incident was opened; the affected products were marked degraded on the public status page.

   Remediation

   • Shifted some traffic to another datacenter
   • Reduced gateway retries to relieve the pressure on the load balancers
   • Stopped the load-balancer processes on the saturated nodes
   • Blocked the retry-triggering requests to the most affected internal endpoint
   • Ramped traffic back up gradually to stabilise the remaining authentication errors

   Action items

   • Correct autoscaling policies so they account for service-mesh sidecar concurrency and capacity, not just the host service
   • Audit request, concurrency and scaling limits for the service mesh across the affected services
   • Review retry and backoff limits across gateways and clients
   • Fix the client retry behavior that amplified authentication traffic
   • Improve load-balancer capacity monitoring
   • Strengthen regional failover safeguards

   Lessons

   GitHub named the core weakness as the service-mesh sidecar that did not scale up, together with its retry behavior, and clients that were not bound enough to keep a partial degradation from amplifying into a broader overload.

   View case
2. IL-0256 Basecamp was fully down twice in two days as hardware load balancers crashed and routers lost sync after failover

   37signals2020

   Symptoms

   On Tuesday 1 September 2020 Basecamp could not be loaded at all for 17 minutes, from 9:51am to 10:08am Central Time (14:51–15:08 UTC), and nothing seemed to work. On Wednesday 2 September it happened again for 15 minutes, from 10:50am to 11:05am Central Time (15:50–16:05 UTC).

   Service outageLink flappingHigh latency

   Trigger

   On 1 September the network links between the data centers went offline, and during the recovery a hardware load balancer crashed. On 2 September the primary load balancer in Virginia crashed again, and later its secondary crashed too.

   Root cause

   Software defect

   Component

   On-premises load balancerBackbone network (WAN)

   Contributing factor

   Overlapping failuresNo headroom

   Remediation

   • On 1 September, intervened by hand to get the network routers working with the standby load balancer
   • On 2 September, fixed the same router and load balancer desynchronization by hand

   Action items

   • Find the cause of the load balancer crash with the vendor and apply its bugfix
   • Replace the hardware load balancers
   • Reroute the redundant cross-datacenter network paths for proper circuit diversity
   • Evaluate moving from hardware to software load balancers, which can be deployed in minutes
   • Evaluate decentralizing the load balancer architecture to limit the impact of one failure
   • Assess the systems for resilience, fragility and risk, and review the assessment process itself

   Lessons

   Basecamp plans for failures like these and practises responses to catastrophic failures in its live systems, but three obscure issues in five days chained into overlapping failures. What caught the team by surprise were cascading failures that exposed unexpected fragility and hard paths to recovery, which Basecamp says it can prepare for.

   View case

By root cause2