Incident Library
Applied filters Root causeFaulty deploymentFactorLimited debug accessClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0315 Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connection
    Railway2025
    Symptoms

    At 18:36 UTC on 28 October 2025, two minutes after a database change went live in production, Railway's internal monitoring alerted and paged the engineering team. The dashboard could not be opened, CLI operations that use the API, such as railway up, failed, GitHub-based deployments were delayed because webhook processing failed, and all Railway Public API operations were disrupted.

    Service outageLock contention or deadlockDatabase overload
    Trigger

    A routine schema change to the backend's Postgres database, introduced at 18:23 UTC and live in production at 18:34 UTC, that added a new column with an index without the CONCURRENTLY option.

    Root cause
    Faulty deployment
    Component
    DatabaseDatabase proxy and connection pooler
    Contributing factor
    No configuration testsLimited debug access
    Detection

    Internal monitoring alerts paged the engineering team at 18:36 UTC, and the incident was declared at 18:41 UTC.

    Remediation
    • The index migration finished after about 30 minutes and released the table lock, and the queued operations and connection pools recovered
    Action items
    • Enforce CONCURRENTLY for all index creation in CI and block pull requests that do not use it
    • Adjust PgBouncer pool limits so they cannot overwhelm the Postgres instance's capacity
    • Configure database user connection limits so administrative access stays possible during incidents
    View case
  2. IL-0591 Gitpod workspaces in all regions became IO bound and unresponsive for days after the ws39 workspace cluster release
    Gitpod2022
    Symptoms

    From April 11, 2022 at 13:17 UTC customers and Gitpod staff reported that Gitpod workspaces were intermittently becoming unresponsive. CPU usage of nodes and pods looked normal, with nothing maxed out, and at 13:58 UTC the team found that the workspaces were IO bound rather than CPU bound.

    Throughput degradationSustained high node CPUNode NotReadyData loss
    Trigger

    The ws39 generation of workspace clusters, released to all regional clusters on April 8th at 20:08 UTC and to the XL clusters on April 9th at 17:45 UTC, split the 1TB disk into smaller RAID0 disks, changed CPU limiting and workspace networking, and moved to faster machine types.

    Root cause
    Faulty deployment
    Component
    Node root diskContainer CPU limit
    Contributing factor
    No staged rolloutGap in automated testsUnrepresentative stagingUnknown limitOverlapping failuresIneffective rollbackHarmful manual actionLimited debug access
    Detection

    Customers and Gitpod staff reported on April 11th at 13:17 UTC that workspaces were intermittently unresponsive, and the team opened an incident in Slack through its incident.io integration. On April 17th the team noticed increased CPU load on the workspace nodes and opened an internal incident.

    Remediation
    • Shifted workspace starts back to the ws38 clusters on April 11th
    • Built ws39a clusters with c2d machines and local SSDs and moved traffic to them
    • Requested a limit increase from Let's Encrypt
    • Routed EU XL traffic to the regional EU cluster and built ws-eu39bxl to replace the crashed ws-eu39xl
    • Cleaned up workspaces stuck in a stopping state so users could start them again
    • Released ws40 with 1TB boot disks to get more IO bandwidth
    • Removed the workspaces that consumed excessive CPU time for crypto-mining
    • Blocked the fraudulent users behind the crypto-mining from using Gitpod
    • Rolled back the ws-daemon change that broke CPU limiting and shipped the fix in the ws34 clusters
    • Blocked ACME HTTP requests to workspace ports so Let's Encrypt rate limits would not be hit again
    • Rolled out IO limiting with the ws41xl and ws41 clusters and moved traffic to them step by step
    Action items
    • Invest in test automation so that test results stop releases like this one
    • Vary the interval between traffic shifts to new clusters, with more time for larger releases
    • Add a risk assessment stage, to be automated, that governs changeset size and rollout speed of workspace cluster rollouts
    • Reduce the impact teams have on each other
    • Add review criteria that find backwards incompatible changes before pull requests are merged, and split such pull requests
    • Recommend rollback as the primary means of mitigation in the incident procedures, and look at automating it
    • Make the status page show the workspace service as down when it is, and update the playbooks
    • Keep incident communication in a single Slack channel and Google Meet
    • Put 10% of the engineering staff on the on-call rotation from May 9th and use that time on observability, alerting and stability
    Lessons

    Gitpod concluded that it should have committed to rolling back on April 11th instead of building and shipping fixes, which kept customers on degraded workspaces longer than necessary. A large release so soon after the previous one needed a risk assessment and slower traffic shifts, and backwards incompatible changes that block a rollback need to be found before pull requests are merged. Failure-free operations require experience with failure.

    View case
  3. IL-0466 Prow scheduled test job pods onto its service cluster, leaving hundreds Pending and starving Prow's own services
    Kubernetes project2019
    Symptoms

    On 8 February 2019 the Kubernetes project's Prow instance at prow.k8s.io became very slow: its controllers took about five minutes to respond to the hook, bot commands such as the cat command stopped working, and the prow.k8s.io site was hard to reach. GitHub status was normal, and the Prow service nodes were found running above 100% of CPU and memory.

    Pods PendingSustained high node CPUHigh latency
    Trigger

    At 13:20 PST an auto-generated bump PR for Prow was merged and deployed, and Prow started scheduling workloads onto the service cluster.

    Root cause
    Faulty deployment
    Component
    CI system
    Contributing factor
    Unrepresentative stagingNo resource requests or limitsLimited debug access
    Detection

    While working on a PR, a maintainer noticed the bot's cat command failing, found GitHub status normal, learned on Slack that the Prow nodes were above 100% CPU and memory, and then found 600 pending pods in the test-pods namespace.

    Remediation
    • Rolled Prow back to v20190206-b01b288 and cleaned up the pods and ProwJobs in the service cluster
    • Applied the revert by hand with kubectl apply after the deployment permission was missing
    • Sent a fix, PR 11209, so the client no longer overwrites a provided literal default client
    • Rolled Prow back to v20190125-2aca69d, from before the bug was introduced
    • Cleaned up the wrongly scheduled pods and ProwJobs again and confirmed Prow was healthy
    Action items
    • Find out why the bug did not show deterministically
    • Be more careful with sweeping Prow changes, with more end-to-end tests or a real staging Prow
    Lessons

    The behaviour was not deterministic, and the change was deployed on a Friday afternoon. Prow instances are set up differently, for example in how build clusters are defined, so another project's Prow running a version fine does not show it is safe for the Kubernetes Prow, which needs a staging environment that looks like it.

    View case
Check your cluster for this

These 3 cases involve CI system and Container CPU limit. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment3 cases

Component
CI system1Container CPU limit1Database1Database proxy and connection pooler1Node root disk1
Contributing factor
Limited debug access3Unrepresentative staging2Unknown limit1Harmful manual action1No configuration tests1No resource requests or limits1No staged rollout1Overlapping failures1Ineffective rollback1Gap in automated tests1
Organisation
Gitpod1Kubernetes project1Railway1
  1. IL-0315 Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connection
    Railway2025
  2. IL-0591 Gitpod workspaces in all regions became IO bound and unresponsive for days after the ws39 workspace cluster release
    Gitpod2022
  3. IL-0466 Prow scheduled test job pods onto its service cluster, leaving hundreds Pending and starving Prow's own services
    Kubernetes project2019