Incident Library
Applied filters Root causeCapacity exhaustionSymptomHigh latencyFactorNo headroomClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases5 cases

1 to 5 of 5

  1. IL-0471 GitHub Actions runs failed or sat queued for 9 hours after a routine deployment's pod replacement saturated capacity
    GitHub2026
    Symptoms

    From 15:05 UTC on 6 August 2026 GitHub Actions workflow runs failed to start, failed partway through or stayed queued for a long time, on both GitHub-hosted and self-hosted runners. Some requests to the Actions REST API returned errors, and some customers saw unexpected rate limiting in their workflows.

    High error rateHigh latency
    Trigger

    A routine deployment to an internal Actions service that processes events and generates Actions jobs replaced pods, and the remaining capacity became saturated.

    Root cause
    Capacity exhaustion
    Component
    CI system
    Contributing factor
    No headroomExcessive blast radiusHarmful manual action
    Remediation
    • Expanded capacity of the affected services
    • Throttled incoming webhook-triggered work
    • Increased processing capacity for the backlog of affected events
    • Deployed changes that stopped runners from repeatedly trying to acquire invalid jobs
    • Deployed a fix for self-hosted runners that were not picking up jobs and enabled it incrementally
    • Paused migrations with GitHub Enterprise Importer
    • Restored webhook-triggered Actions workflows to full throughput
    • Rolled back the mitigation that left some ARC runners offline, and recovered those runners by hand
    • Shared CLI and UI steps for customers to clear jobs that could not be retried or canceled
    Action items
    • Improve deployment and capacity safeguards for the affected services
    • Strengthen monitoring for the conditions that preceded the incident
    • Improve the resiliency and recovery of queued work and runner assignment
    • Add automatic recovery for self-hosted runners in upcoming Runner and ARC releases
    • Reduce the risk of cascading failures and speed up recovery during large Actions disruptions
    View case
  2. IL-0305 Replicas of one Intercom Vitess shard hit 100% CPU and fell up to 2.2 hours behind, serving stale inbox data
    Intercom2026
    Symptoms

    From 17:03 UTC on January 27, 2026, a small subset of Intercom customers, fewer than 1%, saw inconsistent latency and stale data when reading from the platform. At 17:25 UTC an alarm on an excessive message backlog in a worker declared an incident automatically.

    Replication lagDatabase overloadHigh latency
    Trigger

    An internal app deletion process, running more concurrent deletion steps per app than usual with no rate limiting, and the ETL data pipeline both put heavy load on the same database shard at the same time.

    Root cause
    Capacity exhaustion
    Component
    DatabaseDatabase replication
    Contributing factor
    No headroom
    Detection

    An alarm on an excessive message backlog in a worker fired at 17:25 UTC and declared an incident automatically.

    Remediation
    • Disabled ETL processing to reduce load on the affected shard
    • Scaled the app deletion workers to zero
    • Re-enabled ETL carefully once monitoring showed stable metrics
    Action items
    • Increase the capacity of shards to leave headroom for unexpected load spikes
    • Add rate limiting and resource controls to the app deletion process so it backs off when database load is high
    • Review how background work such as ETL and maintenance tasks competes with customer-facing operations for database capacity
    • Alert on replication lag earlier and link it to specific workloads
    • Communicate proactively with affected customers in incidents with a limited blast radius
    Lessons

    Because data is spread across many independent shards, the overload stayed on one shard and reached fewer than 1% of customers instead of the whole platform.

    View case
  3. IL-0341 Traffic between Cloudflare and AWS us-east-1 had high latency, packet loss and origin failures for almost four hours
    Cloudflare2025
    Symptoms

    From 16:27 UTC on 21 August 2025, users connecting to Cloudflare from servers in AWS us-east-1, or receiving connections from Cloudflare there, saw high latency, packet loss and failures to reach origins. Cloudflare's network team was alerted to internal congestion in Ashburn (IAD) at 16:44 UTC.

    Packet lossHigh latencyIntermittent timeouts
    Trigger

    At about 16:27 UTC a single customer began sending many requests from AWS us-east-1 to Cloudflare for objects in Cloudflare's cache, doubling the total traffic from Cloudflare to AWS.

    Root cause
    Capacity exhaustion
    Component
    Transit gateway or network interconnectBGP routingCDN and edge network
    Contributing factor
    No headroomHarmful manual actionExcessive blast radius
    Detection

    Cloudflare's network team was alerted to internal congestion in Ashburn (IAD) at 16:44 UTC, and an incident was raised for customer impact at 17:45 UTC.

    Remediation
    • Rate limited the single customer whose traffic caused the surge
    • Took further traffic engineering actions that fully resolved the congestion
    • Asked AWS to revert its BGP prefix withdrawals, which AWS finished at 20:07 UTC
    Action items
    • Build a mechanism to deprioritise a customer's traffic when it congests the network enough to affect others
    • Speed up the Data Center Interconnect upgrades to add network capacity
    • Work with AWS so the two companies' BGP traffic engineering actions do not conflict
    • Build a traffic management system that gives each customer a network resource budget and automates the manual actions used here
    Lessons

    Cloudflare concludes that it must isolate customers better so that no single customer can take over shared network resources and degrade the platform for others, and that its network needs enough capacity to meet demand. It describes the outage as the result of insufficient network congestion management during an unusual high-traffic event.

    View case
  4. IL-0270 Discord's ScyllaDB authentication cluster overloaded during an OS upgrade, and logins failed for about 50 minutes
    Discord2023
    Symptoms

    On 6 November 2023 at 10:52 AM PST, when an upgrade script drained all connections from zone [c] of the scylla-authentication ScyllaDB cluster, Discord's availability, measured as the share of HTTP requests that return a 200-level response, fell at once to about 25%. Users could not connect, log in, read or send messages.

    Service outageDatabase overloadHigh latency
    Trigger

    During an OS upgrade that takes one zone of a ScyllaDB cluster offline at a time, the upgrade script drained all connections from zone [c] of scylla-authentication at 10:52 AM, while one node in the just-upgraded zone [b] had a degraded NVMe Local SSD.

    Root cause
    Capacity exhaustion
    Component
    Database
    Contributing factor
    No headroomOverlapping failuresDependence on a warm cacheUnverified assumptionIgnored early signalRetry storm
    Detection

    An incident response call had already started when the first alerts fired at 10:56 AM, four minutes after the start. The time spent by downstream service graph showed the authentication service spending far more time on user requests than normal, and the authentication alerts fired moments later.

    Remediation
    • Turned off sending typing events and marking messages as read to shed load from the authentication cluster
    • Waited for the zone [c] RAID rebuild instead of restarting nodes, and the two zone [c] nodes came back online at 11:41 AM
    • Restarted the two zone [b] nodes one at a time
    Action items
    • Double the size of the scylla-authentication cluster so each node takes half as many requests
    • Rebuild scylla-authentication-b-2 with new, non-degraded disks
    • Build a way to detect disk degradation that is too subtle to affect overall latency but still threatens reliability
    • Move upgrade operations into a more automatic control plane for the Scylla clusters that analyses cluster state, such as cache hit rate
    • Increase the redundancy of the scylla-authentication cluster; the options being explored, not yet decided, are a higher replication factor or a hot-standby cluster in another region
    Lessons

    Taking a whole zone out of a three-zone cluster leaves no margin, so one additional slow node in another zone was enough to overload it. For some Scylla clusters the row cache hit rate of restarted nodes must reach a safe level before the next zone goes down, and a short zonal test beforehand gave false confidence. The small but critical authentication cluster needs to be run differently from Discord's much larger Scylla clusters, with more redundancy.

    View case
  5. IL-0383 PagerDuty's shared Cassandra cluster degraded during a repair, delaying notifications and losing incoming events
    PagerDuty2014sev1
    Symptoms

    On 3 June 2014, PagerDuty's notification pipeline performed poorly and some notifications were delayed after the Cassandra cluster behind it became heavily degraded. About 3% of events sent to the integration API could not be received.

    Database overloadHigh error rateHigh latency
    Trigger

    On 3 June 2014 the Cassandra background repair process started its normal run on one node of the cluster while other high load was on the cluster; on 4 June the team restarted the postponed repair by hand.

    Root cause
    Capacity exhaustion
    Component
    Database
    Contributing factor
    No headroomExcessive blast radiusHarmful manual actionIgnored early signal
    Remediation
    • Reduced the load on the Cassandra cluster
    • Stopped the Cassandra repair process
    • Cut communication between some Cassandra nodes to stabilise the cluster
    • On 4 June, deleted all in-flight data in the notification pipeline in a "factory reset"
    • Brought downstream systems back into a consistent state with the emptied pipeline by hand
    Action items
    • Scale the existing Cassandra nodes vertically to bigger, faster servers
    • Set up multiple Cassandra clusters and spread the workloads across them
    • Set system load thresholds at which the Cassandra clusters are scaled out horizontally ahead of need
    • Upgrade the current and new clusters to a more recent Cassandra version
    • Add more load shedding techniques to control Cassandra under high load
    • Bring more Cassandra expertise in-house
    Lessons

    PagerDuty concluded that its Cassandra cluster was underscaled and shared by too many services with different load and usage patterns, and that the scaling work should have come before the planned Cassandra version upgrade.

    View case
Check your cluster for this

These 5 cases involve Database and BGP routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion5 cases

Component
Database3BGP routing1CDN and edge network1CI system1Database replication1Transit gateway or network interconnect1
Contributing factor
No headroom5Excessive blast radius3Harmful manual action3Ignored early signal2Unverified assumption1Overlapping failures1Retry storm1Dependence on a warm cache1
Organisation
Cloudflare1Discord1GitHub1Intercom1PagerDuty1
  1. IL-0471 GitHub Actions runs failed or sat queued for 9 hours after a routine deployment's pod replacement saturated capacity
    GitHub2026
  2. IL-0305 Replicas of one Intercom Vitess shard hit 100% CPU and fell up to 2.2 hours behind, serving stale inbox data
    Intercom2026
  3. IL-0341 Traffic between Cloudflare and AWS us-east-1 had high latency, packet loss and origin failures for almost four hours
    Cloudflare2025
  4. IL-0270 Discord's ScyllaDB authentication cluster overloaded during an OS upgrade, and logins failed for about 50 minutes
    Discord2023
  5. IL-0383 PagerDuty's shared Cassandra cluster degraded during a repair, delaying notifications and losing incoming events
    PagerDuty2014