Cases1 case

1 to 1 of 1

1. IL-0262 Requests through one GitHub datacenter failed or slowed after its load balancer nodes ran out of network flows

   GitHub2026

   Symptoms

   On 17 August 2026 from 13:40 UTC, GitHub issues and pull requests returned errors and slow responses on their pages and in the API. Automated monitoring detected the raised errors, and many other services that route through the same datacenter's load balancers, including the REST and GraphQL APIs, Actions, Copilot, sign-in and authentication, and the webhooks API, also showed errors and latency.

   High error rateHigh latency

   Trigger

   A new peak in traffic pushed the load balancers of one GitHub datacenter past their limits.

   Root cause

   Capacity exhaustion

   Component

   On-premises load balancerService mesh

   Contributing factor

   Retry storm

   Detection

   Automated monitoring detected the elevated errors and an incident was opened; the affected products were marked degraded on the public status page.

   Remediation

   • Shifted some traffic to another datacenter
   • Reduced gateway retries to relieve the pressure on the load balancers
   • Stopped the load-balancer processes on the saturated nodes
   • Blocked the retry-triggering requests to the most affected internal endpoint
   • Ramped traffic back up gradually to stabilise the remaining authentication errors

   Action items

   • Correct autoscaling policies so they account for service-mesh sidecar concurrency and capacity, not just the host service
   • Audit request, concurrency and scaling limits for the service mesh across the affected services
   • Review retry and backoff limits across gateways and clients
   • Fix the client retry behavior that amplified authentication traffic
   • Improve load-balancer capacity monitoring
   • Strengthen regional failover safeguards

   Lessons

   GitHub named the core weakness as the service-mesh sidecar that did not scale up, together with its retry behavior, and clients that were not bound enough to keep a partial degradation from amplifying into a broader overload.

   View case

By root cause1