Cases3 cases

1 to 3 of 3

1. IL-0262 Requests through one GitHub datacenter failed or slowed after its load balancer nodes ran out of network flows

   GitHub2026

   Symptoms

   On 17 August 2026 from 13:40 UTC, GitHub issues and pull requests returned errors and slow responses on their pages and in the API. Automated monitoring detected the raised errors, and many other services that route through the same datacenter's load balancers, including the REST and GraphQL APIs, Actions, Copilot, sign-in and authentication, and the webhooks API, also showed errors and latency.

   High error rateHigh latency

   Trigger

   A new peak in traffic pushed the load balancers of one GitHub datacenter past their limits.

   Root cause

   Capacity exhaustion

   Component

   On-premises load balancerService mesh

   Contributing factor

   Retry storm

   Detection

   Automated monitoring detected the elevated errors and an incident was opened; the affected products were marked degraded on the public status page.

   Remediation

   • Shifted some traffic to another datacenter
   • Reduced gateway retries to relieve the pressure on the load balancers
   • Stopped the load-balancer processes on the saturated nodes
   • Blocked the retry-triggering requests to the most affected internal endpoint
   • Ramped traffic back up gradually to stabilise the remaining authentication errors

   Action items

   • Correct autoscaling policies so they account for service-mesh sidecar concurrency and capacity, not just the host service
   • Audit request, concurrency and scaling limits for the service mesh across the affected services
   • Review retry and backoff limits across gateways and clients
   • Fix the client retry behavior that amplified authentication traffic
   • Improve load-balancer capacity monitoring
   • Strengthen regional failover safeguards

   Lessons

   GitHub named the core weakness as the service-mesh sidecar that did not scale up, together with its retry behavior, and clients that were not bound enough to keep a partial degradation from amplifying into a broader overload.

   View case
2. IL-0471 GitHub Actions runs failed or sat queued for 9 hours after a routine deployment's pod replacement saturated capacity

   GitHub2026

   Symptoms

   From 15:05 UTC on 6 August 2026 GitHub Actions workflow runs failed to start, failed partway through or stayed queued for a long time, on both GitHub-hosted and self-hosted runners. Some requests to the Actions REST API returned errors, and some customers saw unexpected rate limiting in their workflows.

   High error rateHigh latency

   Trigger

   A routine deployment to an internal Actions service that processes events and generates Actions jobs replaced pods, and the remaining capacity became saturated.

   Root cause

   Capacity exhaustion

   Component

   CI system

   Contributing factor

   No headroomExcessive blast radiusHarmful manual action

   Remediation

   • Expanded capacity of the affected services
   • Throttled incoming webhook-triggered work
   • Increased processing capacity for the backlog of affected events
   • Deployed changes that stopped runners from repeatedly trying to acquire invalid jobs
   • Deployed a fix for self-hosted runners that were not picking up jobs and enabled it incrementally
   • Paused migrations with GitHub Enterprise Importer
   • Restored webhook-triggered Actions workflows to full throughput
   • Rolled back the mitigation that left some ARC runners offline, and recovered those runners by hand
   • Shared CLI and UI steps for customers to clear jobs that could not be retried or canceled

   Action items

   • Improve deployment and capacity safeguards for the affected services
   • Strengthen monitoring for the conditions that preceded the incident
   • Improve the resiliency and recovery of queued work and runner assignment
   • Add automatic recovery for self-hosted runners in upcoming Runner and ARC releases
   • Reduce the risk of cascading failures and speed up recovery during large Actions disruptions

   View case
3. IL-0302 Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022

   GitHub2022

   Symptoms

   On March 16, 2022 at 14:09 UTC, write operations across GitHub stopped working, including git operations, webhooks, pull requests, API requests, issues, GitHub Packages, GitHub Codespaces, GitHub Actions and GitHub Pages. Under peak-hour load, the database proxy in front of the shared mysql1 MySQL database had reached its maximum number of connections.

   Writes rejectedDatabase overloadHigh error rateFailover failure

   Trigger

   Increased load on mysql1 during peak hours, combined with poor query performance in specific circumstances, pushed the database proxy to its maximum number of connections.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Inefficient queryChange during peak hoursHarmful manual actionExcessive blast radius

   Remediation

   • Failed over mysql1 to a healthy replica on March 16
   • Reset the application connections to the new primary on March 17
   • Added an index that fixed the main query performance problem
   • Performed a primary failover on March 22
   • Performed a primary failover on March 23
   • Throttled webhook traffic to reduce load

   Action items

   • Keep throttling webhook traffic at peak load times
   • Audit the load patterns of mysql1 during peak hours and make performance fixes based on the audits
   • Move traffic to other databases to reduce load and speed up failover
   • Review change management, particularly monitoring and changes during high load in production
   • Shard the databases and scale up hardware

   View case

By root cause1