Incident Library

IL-0471 GitHub · 2026-08-06kubernetes

GitHub Actions runs failed or sat queued for 9 hours after a routine deployment's pod replacement saturated capacity

On 6 August 2026 a routine deployment to an internal GitHub Actions service that processes events and generates Actions jobs exposed a capacity and concurrency weakness that already existed. As the deployment replaced pods, the remaining capacity saturated, services crashed, and the failure cascaded across multiple clusters and downstream services. These services recovered at 17:00 UTC after capacity was expanded, incoming webhook-triggered work was throttled, and processing capacity for the backlog of affected events was increased.

By then a backlog had built up in the systems that assign jobs to runners. A latent bug in one of the job assignment services gave runners jobs that were no longer valid, and the runners got stuck retrying them instead of taking valid work. GitHub deployed changes that stopped runners from repeatedly trying to acquire invalid jobs and a fix for self-hosted runners that were not picking up jobs, enabled incrementally. The queues then drained, webhook-triggered workflows returned to full throughput, and Actions was back to normal at 00:14 UTC on 7 August, 9h 9m after the start. Migrations with GitHub Enterprise Importer were paused during mitigation.

One mitigation deployed during the incident inadvertently affected some Actions Runner Controller (ARC) runners, which stayed offline until they were recovered by hand; GitHub later rolled that change back and told affected users to delete the stuck idle pods with kubectl or redeploy ARC so it would create replacement runners. For jobs left stuck without retry or cancel, GitHub shared CLI and UI steps in a community discussion, and customers might need to repeat the push or pull request actions whose events were never processed.

Severity
–
Scope
GitHub Actions workflow runs on GitHub-hosted and self-hosted runners: at the peak 71% had infrastructure failures and 75% of the rest were delayed by more than 5 minutes; GitHub Pages, Copilot code review, Copilot coding agent and GitHub Enterprise Importer migrations were also affected
Users affected
Customers using GitHub-hosted and self-hosted runners
Region
–
Time to detect
–
Time to resolve
9h 9m

Symptoms

From 15:05 UTC on 6 August 2026 GitHub Actions workflow runs failed to start, failed partway through or stayed queued for a long time, on both GitHub-hosted and self-hosted runners. Some requests to the Actions REST API returned errors, and some customers saw unexpected rate limiting in their workflows.

At the peak, 71% of workflow runs had infrastructure failures and 75% of the remaining runs were delayed by more than 5 minutes. Status updates gave job success rates of 30 to 40% at the low point, about 65% at 20:34 UTC, 97% at 22:18 UTC and 99% at 23:13 UTC. Self-hosted runners saw errors or rate limiting when registering. Copilot code review, Copilot coding agent, GitHub Pages and migrations with GitHub Enterprise Importer were affected as well, and webhook deliveries were delayed.

To help recovery, webhook triggers were throttled, at one point to about 15% of webhooks, so many push and pull request events did not start workflow runs, and some of those events were never processed and could not be replayed automatically. Later in the incident, runners were stuck retrying jobs that were no longer valid instead of picking up valid work. Afterwards some Actions Runner Controller (ARC) runner pods stayed stuck in an idle state, and some jobs created during the incident could neither be retried nor canceled.

Symptom High error rateHigh latency

Root cause

Root cause Capacity exhaustion

Component CI system
Contributing factor No headroomExcessive blast radiusHarmful manual action
Trigger
A routine deployment to an internal Actions service that processes events and generates Actions jobs replaced pods, and the remaining capacity became saturated.

Detection

–

Remediation

Action items

Lessons

–

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, CI system among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.