Incident Library
Applied filters Root causeCapacity exhaustionComponentCI systemDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0471 GitHub Actions runs failed or sat queued for 9 hours after a routine deployment's pod replacement saturated capacity
    GitHub2026
    Symptoms

    From 15:05 UTC on 6 August 2026 GitHub Actions workflow runs failed to start, failed partway through or stayed queued for a long time, on both GitHub-hosted and self-hosted runners. Some requests to the Actions REST API returned errors, and some customers saw unexpected rate limiting in their workflows.

    High error rateHigh latency
    Trigger

    A routine deployment to an internal Actions service that processes events and generates Actions jobs replaced pods, and the remaining capacity became saturated.

    Root cause
    Capacity exhaustion
    Component
    CI system
    Contributing factor
    No headroomExcessive blast radiusHarmful manual action
    Remediation
    • Expanded capacity of the affected services
    • Throttled incoming webhook-triggered work
    • Increased processing capacity for the backlog of affected events
    • Deployed changes that stopped runners from repeatedly trying to acquire invalid jobs
    • Deployed a fix for self-hosted runners that were not picking up jobs and enabled it incrementally
    • Paused migrations with GitHub Enterprise Importer
    • Restored webhook-triggered Actions workflows to full throughput
    • Rolled back the mitigation that left some ARC runners offline, and recovered those runners by hand
    • Shared CLI and UI steps for customers to clear jobs that could not be retried or canceled
    Action items
    • Improve deployment and capacity safeguards for the affected services
    • Strengthen monitoring for the conditions that preceded the incident
    • Improve the resiliency and recovery of queued work and runner assignment
    • Add automatic recovery for self-hosted runners in upcoming Runner and ARC releases
    • Reduce the risk of cascading failures and speed up recovery during large Actions disruptions
    View case
  2. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
    Symptoms

    Builds on Zalando's in-house Continuous Delivery Platform (CDP), which runs CI/CD on Kubernetes, timed out and failed for almost an hour on 11 June 2019. The Kubernetes builder pods could not get AWS IAM credentials.

    High error rateCredentials unavailableControl plane lag
    Trigger

    An unlucky rescaling of the CDP Kubernetes cluster left only one node available for builder pods. Builder pods were created and deleted on that node so quickly that kubelet fell behind.

    Root cause
    Capacity exhaustion
    Component
    kubeletCredential broker for podsKubernetes API serverCI system
    Contributing factor
    Unreviewed defaultsSingle point of failure
    Remediation
    • Scaled the CDP cluster up by hand so builder pods spread over more than one node
    Action items
    • Raise the kubelet QPS limit from the default 5 to 50 in the global cluster configuration
    View case
  3. IL-0401 Prow jobs on the k8s-prow-build GKE cluster failed and merges stalled as nodes hit IOPS throttling and kernel panics
    Kubernetes project2019
    Symptoms

    On 30 May 2019, in the run-up to the Kubernetes 1.15 code freeze, Prow test jobs in the k8s-prow-build cluster failed in surprising ways or were aborted partway through. Jobs were reported to GitHub as failed while Deck still showed them as running, because pods exited without uploading `finished.json`, the file most Prow tooling uses to tell a job's state.

    Node NotReadyNode kernel panicPods stuck in TerminatingThroughput degradation
    Trigger

    A spike in test load before the Kubernetes 1.15 code freeze pushed disk I/O on the build nodes far beyond the IOPS that GCP allows for 250 GB disks.

    Root cause
    Capacity exhaustion
    Component
    Cloud block storageNode root diskkubeletCI system
    Contributing factor
    Monitoring blind spotIgnored early signal
    Detection

    Several users reported jobs that GitHub showed as failed but Deck showed as still running, and the first report came on Slack at 13:26. Alerting for these failures did not exist or did not work.

    Remediation
    • Requested and received additional disk quota
    • Brought up 55 nodes with 500 GB disks in a new node pool
    • Migrated all nodes from the 250 GB disk pool to the 500 GB disk pool by draining old nodes, finishing at 23:55
    Action items
    • Create or fix monitoring so the oncall is paged when tide stops doing things
    • Set a process and guidelines for adding new members to the oncall rotation, with preparation and extra support
    Lessons

    Prow metrics were not recorded before the incident, which made it hard to understand, and alerting for these failures did not exist or did not work. After the cause was found and a fix devised, nothing happened for another two hours. The team counts it as luck that someone who recognised IOPS as the problem happened to be around to look.

    View case
Check your cluster for this

These 3 cases involve CI system and kubelet. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion3 cases

Component
CI system3kubelet2Cloud block storage1Credential broker for pods1Kubernetes API server1Node root disk1
Contributing factor
Excessive blast radius1Unreviewed defaults1Ignored early signal1Harmful manual action1Monitoring blind spot1No headroom1Single point of failure1
Organisation
GitHub1Kubernetes project1Zalando1
  1. IL-0471 GitHub Actions runs failed or sat queued for 9 hours after a routine deployment's pod replacement saturated capacity
    GitHub2026
  2. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
  3. IL-0401 Prow jobs on the k8s-prow-build GKE cluster failed and merges stalled as nodes hit IOPS throttling and kernel panics
    Kubernetes project2019