Incident Library
Applied filters ComponentService meshSymptomHigh latencyFactorRetry stormClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0262 Requests through one GitHub datacenter failed or slowed after its load balancer nodes ran out of network flows
    GitHub2026
    Symptoms

    On 17 August 2026 from 13:40 UTC, GitHub issues and pull requests returned errors and slow responses on their pages and in the API. Automated monitoring detected the raised errors, and many other services that route through the same datacenter's load balancers, including the REST and GraphQL APIs, Actions, Copilot, sign-in and authentication, and the webhooks API, also showed errors and latency.

    High error rateHigh latency
    Trigger

    A new peak in traffic pushed the load balancers of one GitHub datacenter past their limits.

    Root cause
    Capacity exhaustion
    Component
    On-premises load balancerService mesh
    Contributing factor
    Retry storm
    Detection

    Automated monitoring detected the elevated errors and an incident was opened; the affected products were marked degraded on the public status page.

    Remediation
    • Shifted some traffic to another datacenter
    • Reduced gateway retries to relieve the pressure on the load balancers
    • Stopped the load-balancer processes on the saturated nodes
    • Blocked the retry-triggering requests to the most affected internal endpoint
    • Ramped traffic back up gradually to stabilise the remaining authentication errors
    Action items
    • Correct autoscaling policies so they account for service-mesh sidecar concurrency and capacity, not just the host service
    • Audit request, concurrency and scaling limits for the service mesh across the affected services
    • Review retry and backoff limits across gateways and clients
    • Fix the client retry behavior that amplified authentication traffic
    • Improve load-balancer capacity monitoring
    • Strengthen regional failover safeguards
    Lessons

    GitHub named the core weakness as the service-mesh sidecar that did not scale up, together with its retry behavior, and clients that were not bound enough to keep a partial degradation from amplifying into a broader overload.

    View case
Check your cluster for this

This case involves On-premises load balancer and Service mesh. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
On-premises load balancerService mesh
Contributing factor
Retry storm
Organisation
GitHub
  1. IL-0262 Requests through one GitHub datacenter failed or slowed after its load balancer nodes ran out of network flows
    GitHub2026