Incident Library

Applied filters Root causeCapacity exhaustionSymptomHigh error rateFactorNo headroomClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

Cases3 cases

1 to 3 of 3

IL-0471 GitHub Actions runs failed or sat queued for 9 hours after a routine deployment's pod replacement saturated capacity
GitHub2026
Symptoms
From 15:05 UTC on 6 August 2026 GitHub Actions workflow runs failed to start, failed partway through or stayed queued for a long time, on both GitHub-hosted and self-hosted runners. Some requests to the Actions REST API returned errors, and some customers saw unexpected rate limiting in their workflows.
High error rate High latency
Trigger
A routine deployment to an internal Actions service that processes events and generates Actions jobs replaced pods, and the remaining capacity became saturated.
Root cause
Capacity exhaustion
Component
CI system
Contributing factor
No headroom Excessive blast radius Harmful manual action
Remediation
Expanded capacity of the affected services
Throttled incoming webhook-triggered work
Increased processing capacity for the backlog of affected events
Deployed changes that stopped runners from repeatedly trying to acquire invalid jobs
Deployed a fix for self-hosted runners that were not picking up jobs and enabled it incrementally
Paused migrations with GitHub Enterprise Importer
Restored webhook-triggered Actions workflows to full throughput
Rolled back the mitigation that left some ARC runners offline, and recovered those runners by hand
Shared CLI and UI steps for customers to clear jobs that could not be retried or canceled
Action items
Improve deployment and capacity safeguards for the affected services
Strengthen monitoring for the conditions that preceded the incident
Improve the resiliency and recovery of queued work and runner assignment
Add automatic recovery for self-hosted runners in upcoming Runner and ARC releases
Reduce the risk of cascading failures and speed up recovery during large Actions disruptions
View case
IL-0465 IONOS Container Registry image pushes and pulls failed with 504 Gateway Timeout
IONOS Cloud2026
Symptoms
On 4 March 2026 customers of the IONOS Container Registry got 504 Gateway Timeout errors when pushing or pulling container images, and deployments that relied on the registry were blocked. The status page first reported an increased error count on the registry at 09:23 UTC, saying customers might be unable to pull images.
Image pull failures High error rate
Trigger
A migration combined with a temporary capacity constraint placed two critical control plane components on the same proxy instance, despite anti-affinity rules; the shared proxy then reached its concurrent connection limit.
Root cause
Capacity exhaustion
Component
Container registry and its credentials Managed Kubernetes control plane and provider API Pod scheduling and placement
Contributing factor
No headroom Single point of failure No alert on limit
Remediation
Rolled back a component of the Kubernetes control plane to an earlier version
Provisioned additional proxy capacity
Moved the affected control plane components onto separate proxy instances
Action items
Redesign registry-to-API connectivity so each node uses its own local proxy
Adjust alert thresholds to warn before proxy connection use approaches capacity
Deploy more infrastructure clusters and redistribute registries so no cluster exceeds safe capacity, with automation that keeps balancing load
View case
IL-0383 PagerDuty's shared Cassandra cluster degraded during a repair, delaying notifications and losing incoming events
PagerDuty2014sev1
Symptoms
On 3 June 2014, PagerDuty's notification pipeline performed poorly and some notifications were delayed after the Cassandra cluster behind it became heavily degraded. About 3% of events sent to the integration API could not be received.
Database overload High error rate High latency
Trigger
On 3 June 2014 the Cassandra background repair process started its normal run on one node of the cluster while other high load was on the cluster; on 4 June the team restarted the postponed repair by hand.
Root cause
Capacity exhaustion
Component
Database
Contributing factor
No headroom Excessive blast radius Harmful manual action Ignored early signal
Remediation
Reduced the load on the Cassandra cluster
Stopped the Cassandra repair process
Cut communication between some Cassandra nodes to stabilise the cluster
On 4 June, deleted all in-flight data in the notification pipeline in a "factory reset"
Brought downstream systems back into a consistent state with the emptied pipeline by hand
Action items
Scale the existing Cassandra nodes vertically to bigger, faster servers
Set up multiple Cassandra clusters and spread the workloads across them
Set system load thresholds at which the Cassandra clusters are scaled out horizontally ahead of need
Upgrade the current and new clusters to a more recent Cassandra version
Add more load shedding techniques to control Cassandra under high load
Bring more Cassandra expertise in-house
Lessons
PagerDuty concluded that its Cassandra cluster was underscaled and shared by too many services with different load and usage patterns, and that the scaling work should have come before the planned Cassandra version upgrade.
View case

Check your cluster for this

These 3 cases involve CI system and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

Graph

Cases3 cases

By root cause1

Capacity exhaustion3 cases