Incident Library

IL-0262 GitHub · 2026-08-17networking

Requests through one GitHub datacenter failed or slowed after its load balancer nodes ran out of network flows

A new traffic peak pushed one GitHub datacenter's load balancers towards and then past their limits. A service-mesh sidecar hit its concurrency limit and failed to scale up; GitHub's follow-up is to make autoscaling policies account for the sidecar's concurrency and capacity, not just the host service. As requests backed up, several load-balancer nodes exhausted their network flow limits, which degraded the shared gateway authentication path. Issues, pull requests, the APIs, Actions, Copilot and other services routed through that datacenter began returning errors and slow responses. A latent client retry bug amplified traffic to one internal authentication endpoint, slowing recovery for the Copilot Token Service; GitHub names its retry behavior, together with clients that were not bound enough, as what let the partial degradation grow into a broader overload.

Automated monitoring opened an incident and the affected products were marked degraded on the public status page. Engineers traced the failure to network saturation on the datacenter's load balancers, shifted some traffic to another datacenter and reduced gateway retries. Stopping the load-balancer processes on the saturated nodes and blocking the retry-triggering requests to the most affected internal endpoint brought broad and immediate recovery. Remaining authentication errors from client retry amplification were stabilised by ramping traffic back up gradually, and the incident was resolved after a sustained period of healthy telemetry. No data was lost. The report gives the incident as lasting 7 hours and 35 minutes, with impact for approximately 6 hours 44 minutes.

Severity
–
Scope
Issues and pull requests (pages and API) and other services routed through the same datacenter's load balancers: REST and GraphQL APIs, Actions, Copilot, sign-in and authentication, webhooks API; impact for approximately 6 hours 44 minutes; no data lost
Users affected
about 29K organizations saw at least one failed or slow request; over roughly 4.8M failed or slow requests in total
Region
one GitHub datacenter
Time to detect
–
Time to resolve
7h 35m

Symptoms

On 17 August 2026 from 13:40 UTC, GitHub issues and pull requests returned errors and slow responses on their pages and in the API. Automated monitoring detected the raised errors, and many other services that route through the same datacenter's load balancers, including the REST and GraphQL APIs, Actions, Copilot, sign-in and authentication, and the webhooks API, also showed errors and latency.

At the peak, 56.07% of front-door requests to the affected services failed or were slow at the edge. Across the window about 29K organizations saw at least one failed or slow request, over roughly 4.8M failed or slow requests in total. Retrying often worked, but during the worst stretches a large share of attempts failed.

The overload started in one datacenter when a new traffic peak pushed its load balancers past their limits. A service-mesh sidecar reached its concurrency limit and did not scale up, requests backed up, and several of the datacenter's load-balancer nodes exhausted their network flow limits. That degraded the shared gateway authentication path, so authentication became slow and failed across the many services behind that datacenter. A latent client retry bug then sharply raised traffic to one internal authentication endpoint and slowed recovery of the Copilot Token Service.

Engineers traced the failure to network saturation on the load balancers of the single datacenter. They began shifting some traffic to another datacenter and reducing gateway retries to relieve the pressure. Broad and immediate recovery came when the load-balancer processes on the saturated nodes were stopped and the requests that triggered retries to the most affected internal endpoint were blocked.

Symptom High error rateHigh latency

Root cause

Root cause Capacity exhaustion

Component On-premises load balancerService mesh
Contributing factor Retry storm
Trigger
A new peak in traffic pushed the load balancers of one GitHub datacenter past their limits.

Detection

Automated monitoring detected the elevated errors and an incident was opened; the affected products were marked degraded on the public status page.

Remediation

Action items

Lessons

GitHub named the core weakness as the service-mesh sidecar that did not scale up, together with its retry behavior, and clients that were not bound enough to keep a partial degradation from amplifying into a broader overload.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, On-premises load balancer, Service mesh among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.