Cases3 cases

1 to 3 of 3

1. IL-0262 Requests through one GitHub datacenter failed or slowed after its load balancer nodes ran out of network flows

   GitHub2026

   Symptoms

   On 17 August 2026 from 13:40 UTC, GitHub issues and pull requests returned errors and slow responses on their pages and in the API. Automated monitoring detected the raised errors, and many other services that route through the same datacenter's load balancers, including the REST and GraphQL APIs, Actions, Copilot, sign-in and authentication, and the webhooks API, also showed errors and latency.

   High error rateHigh latency

   Trigger

   A new peak in traffic pushed the load balancers of one GitHub datacenter past their limits.

   Root cause

   Capacity exhaustion

   Component

   On-premises load balancerService mesh

   Contributing factor

   Retry storm

   Detection

   Automated monitoring detected the elevated errors and an incident was opened; the affected products were marked degraded on the public status page.

   Remediation

   • Shifted some traffic to another datacenter
   • Reduced gateway retries to relieve the pressure on the load balancers
   • Stopped the load-balancer processes on the saturated nodes
   • Blocked the retry-triggering requests to the most affected internal endpoint
   • Ramped traffic back up gradually to stabilise the remaining authentication errors

   Action items

   • Correct autoscaling policies so they account for service-mesh sidecar concurrency and capacity, not just the host service
   • Audit request, concurrency and scaling limits for the service mesh across the affected services
   • Review retry and backoff limits across gateways and clients
   • Fix the client retry behavior that amplified authentication traffic
   • Improve load-balancer capacity monitoring
   • Strengthen regional failover safeguards

   Lessons

   GitHub named the core weakness as the service-mesh sidecar that did not scale up, together with its retry behavior, and clients that were not bound enough to keep a partial degradation from amplifying into a broader overload.

   View case
2. IL-0455 Calls from an AWS service through NAT Gateway to an on-premises service returned 5xx errors every busy weekend

   Woowa Brothers2020

   Symptoms

   5xx errors rose on API requests from service A, running in AWS, to service B, running on premises, and user requests failed. At the same time the L4 session count of service B went up.

   High error rateOutbound connection exhaustion

   Trigger

   High weekend traffic from service A, which called service B without a connection pool, through the NAT gateways.

   Root cause

   Capacity exhaustion

   Component

   NAT gatewayOn-premises load balancer

   Contributing factor

   Unknown limit

   Detection

   Rising 5xx errors on service A and a rising L4 session count on service B.

   Remediation

   • Added service A instances, which lowered but did not end the errors
   • Replaced the service A instances, which brought errors and the L4 session count back to normal

   Action items

   • Switch service A to a connection pool
   • Find the services that use NAT Gateway between AWS and the IDC and switch them to connection pools

   Lessons

   The fix is often easy to find once the cause is analysed, but without confirming the root cause a service keeps running with the risk still in it, so Woowa Brothers treats finding the root cause as essential for preventing recurrence, alongside resolving the failure first.

   View case
3. IL-0492 Hosted Graphite's self-hosted load balancers hit their connection limit in an AWS outage, timing out its site and APIs

   Hosted Graphite2018

   Symptoms

   On a Friday night at 23:28 UTC, Hosted Graphite's monitoring showed its website intermittently becoming unavailable: periodic checks of the website and of the graph rendering API were timing out. Traffic dropped across all ingestion protocols at the same time.

   Intermittent timeoutsIntermittent connection failures

   Trigger

   An AWS connectivity problem in us-east1 and us-west2 slowed down connections from AWS hosts in those regions to Hosted Graphite's load balancers without breaking them.

   Root cause

   Capacity exhaustion

   Component

   On-premises load balancer

   Contributing factor

   Circular dependency

   Detection

   Monitoring reported the website intermittently unavailable at 23:28 UTC, when periodic checks of the website and the graph rendering API started timing out.

   Action items

   • Enforce stricter client timeouts
   • Raise the load balancers' connection limits
   • Use the new visualizations built from load balancer logs when troubleshooting the load balancing layer

   Lessons

   An outage at a provider that did not host the service's website still made the service unavailable, here because slow clients in the affected regions used up the connection limit of the load balancers that served everyone.

   View case

By root cause1