Incident Library

Applied filters SymptomHigh latencyFactorHarmful manual actionOrganisationGitHubClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

Cases3 cases

1 to 3 of 3

IL-0471 GitHub Actions runs failed or sat queued for 9 hours after a routine deployment's pod replacement saturated capacity
GitHub2026
Symptoms
From 15:05 UTC on 6 August 2026 GitHub Actions workflow runs failed to start, failed partway through or stayed queued for a long time, on both GitHub-hosted and self-hosted runners. Some requests to the Actions REST API returned errors, and some customers saw unexpected rate limiting in their workflows.
High error rate High latency
Trigger
A routine deployment to an internal Actions service that processes events and generates Actions jobs replaced pods, and the remaining capacity became saturated.
Root cause
Capacity exhaustion
Component
CI system
Contributing factor
No headroom Excessive blast radius Harmful manual action
Remediation
Expanded capacity of the affected services
Throttled incoming webhook-triggered work
Increased processing capacity for the backlog of affected events
Deployed changes that stopped runners from repeatedly trying to acquire invalid jobs
Deployed a fix for self-hosted runners that were not picking up jobs and enabled it incrementally
Paused migrations with GitHub Enterprise Importer
Restored webhook-triggered Actions workflows to full throughput
Rolled back the mitigation that left some ARC runners offline, and recovered those runners by hand
Shared CLI and UI steps for customers to clear jobs that could not be retried or canceled
Action items
Improve deployment and capacity safeguards for the affected services
Strengthen monitoring for the conditions that preceded the incident
Improve the resiliency and recovery of queued work and runner assignment
Add automatic recovery for self-hosted runners in upcoming Runner and ARC releases
Reduce the risk of cascading failures and speed up recovery during large Actions disruptions
View case
IL-0263 GitHub was slow and erroring all day and down for 18 minutes as links between access and aggregation switches saturated
GitHub2012
Symptoms
On Friday, 30 November 2012, GitHub had sporadic bursts of slow responses and intermittent errors for the whole day and was completely unavailable for 18 minutes. One fileserver pair was offline for a large part of the day, which affected a small percentage of repositories; this was a side effect of the network problems on the high-availability clustering between the fileserver nodes.
High latency High error rate Service outage
Trigger
On the morning of 30 November GitHub moved the A switches in its server cabinets onto the new aggregation switches, the second half of a migration whose B-switch half had been done the day before.
Root cause
Software defect
Component
Network switch NIC, cable and switch port link
Contributing factor
Overlapping failures Unverified assumption Harmful manual action
Detection
Engineers saw the problems during the maintenance itself: connectivity to the B switch was lost as the last A switch was connected, and afterwards performance stayed low and the links between access and aggregation switches showed as completely saturated.
Remediation
Fixed the bridge-loop problem and returned the affected B switch to service
Removed the partial link failure detection setting from the access switches
Escalated to the network vendor, which raised the case to its highest severity level
Restarted core processes on the aggregation switches in a Saturday 0600 Pacific maintenance window so they learned MAC addresses again
Action items
Deploy the vendor's software update that fixes the MAC learning issue to the switches
Closely monitor aggregation-to-access capacity and keep a workaround process ready if the problem returns
Build a duplicate of the network stack, from the routers through the access switches, as a staging environment for testing such changes
Add automated network monitoring that alerts sooner on similar issues
Run scheduled incident response exercises with scenarios that counter tunnel vision
Contact the network vendor's support at the first sign of trouble in the network
Lessons
GitHub had designed the maintenance to have no impact on customers and clearly failed, so it plans to invest in a staging copy of its network to test such changes and catch bugs like this one. During the response the team fixated for a long time on a bridge loop, which kept it from considering other causes. GitHub had a very positive experience with the vendor's support staff, so it will contact that support team at the first sign of network trouble.
View case
IL-0303 GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went down
GitHub2012
Symptoms
On Monday, September 10th, 2012, during a schema migration, the health checks on GitHub's MySQL master failed and Percona Replication Manager moved the 'active' role to another node of the new three-node cluster. That node's InnoDB buffer pool was cold and it performed poorly, its health checks failed in turn, and the role failed back to the original server. GitHub.com went down.
Failover failure Database overload High latency Replication lag Split brain Service outage
Trigger
A schema migration on Monday, September 10th put more load on the master database than earlier migrations had, and Percona Replication Manager's health checks on the master failed.
Root cause
Software defect
Component
Database Database failover manager Database replication
Contributing factor
Automation amplification Dependence on a warm cache Harmful manual action Overlapping failures
Detection
On the morning of September 11th, a developer told the operations team that the 'standby' node was returning incorrect query results.
Remediation
Enabled Pacemaker's maintenance-mode so no health checks or automatic failovers ran, and let the buffer pool warm
Powered off the out-of-date node 'c' to end the partition
Stopped Pacemaker and Heartbeat on nodes 'a' and 'b' and started them on node 'a' only
Started MySQL on node 'a' in the active role
Removed the events leaked to wrong dashboards and audited the wrongly routed repositories
Contacted the owners of the exposed private repositories
Action items
Change the Pacemaker configuration so the active database role fails over only when an operations engineer starts it
Find a way to warm the InnoDB buffer pool of slave databases so failovers do not hurt performance
Audit the Pacemaker and Heartbeat stack, focusing on the code path behind the segfault
Run hardware tests on the server where the segfault happened
Lessons
GitHub concluded that automated failover is not the right strategy for its primary production database: for every failover in this incident, any member of the operations team would have said it should not happen. Failovers that must happen, in an emergency or during maintenance, should not hurt performance.
View case

Check your cluster for this

These 3 cases involve CI system and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

Graph

Cases3 cases

By root cause2

Capacity exhaustion1 case

Software defect2 cases