IL-0466 Kubernetes project · 2019-02-08kubernetesgcp
Prow scheduled test job pods onto its service cluster, leaving hundreds Pending and starving Prow's own services
Prow, the Kubernetes project's CI system, runs its controllers in a service cluster and schedules test jobs into separate build clusters. The build-cluster secret literally defines a "default" cluster alias pointing at the build cluster. A client refactor in Prow did not respect that literal alias and fell back to its own default, the service cluster. After a Prow bump was deployed at 13:20 PST on 8 February 2019, test jobs were scheduled into the service cluster, where they could not start because their volumes and credentials were missing, and the service cluster ran out of resources. In the chat, a responder who was surprised that the scheduler could starve the core Prow services asked whether CPU and memory requests and limits were in use, and from the reply took it that no Prow service deployment had them.
Prow was first noticed slow at 16:04. At 16:40 Prow was rolled back to v20190206-b01b288 and the pods and ProwJobs in the service cluster were cleaned up, but the responder lacked permission for the manual deployment that was needed, filed an issue and applied the revert by hand with kubectl apply at 16:50. The problem persisted because the bug was older than that bump and did not show deterministically. At 17:15 the literal "default" alias was identified, a fix was sent as PR 11209, and Prow was rolled back to v20190125-2aca69d. By 17:59 the wrong pods and ProwJobs were cleaned up again and Prow was healthy; the mailing list was notified at 18:09.
- Severity
- –
- Scope
- Test jobs scheduled onto the service cluster could not start from 1:46pm to 5:30pm PST, and all other Prow controllers were very slow or unusable from 4:06pm to 5:30pm
- Users affected
- –
- Region
- –
- Time to detect
- 2h 18m
- Time to resolve
- 3h 44m
Symptoms
On 8 February 2019 the Kubernetes project's Prow instance at prow.k8s.io became very slow: its controllers took about five minutes to respond to the hook, bot commands such as the cat command stopped working, and the prow.k8s.io site was hard to reach. GitHub status was normal, and the Prow service nodes were found running above 100% of CPU and memory.
kubectl top showed a node of the service cluster at "gke-prow-default-pool-116f5d3e-bl6h 4022m 102% 13681Mi 110%". The test-pods namespace of the service cluster held hundreds of test job pods that belonged in the build cluster, 600 Pending pods stuck because they could not find the right secret, and the GKE console showed 225 unschedulable pods. Jobs that did run failed on network operations and on uploading logs to GCS, and /lgtm and assignment commands returned errors.
Rolling Prow back to the version before that day's bump, v20190206-b01b288, did not stop the problem. The bug was found when a responder asked whether the cluster file literally defined a "default" cluster alias, and a rollback to v20190125-2aca69d, from before the bug was introduced, ended it.
Root cause
Root cause Faulty deployment
- Trigger
- At 13:20 PST an auto-generated bump PR for Prow was merged and deployed, and Prow started scheduling workloads onto the service cluster.
Detection
While working on a PR, a maintainer noticed the bot's cat command failing, found GitHub status normal, learned on Slack that the Prow nodes were above 100% CPU and memory, and then found 600 pending pods in the test-pods namespace.
Remediation
- Rolled Prow back to v20190206-b01b288 and cleaned up the pods and ProwJobs in the service cluster
- Applied the revert by hand with kubectl apply after the deployment permission was missing
- Sent a fix, PR 11209, so the client no longer overwrites a provided literal default client
- Rolled Prow back to v20190125-2aca69d, from before the bug was introduced
- Cleaned up the wrongly scheduled pods and ProwJobs again and confirmed Prow was healthy
Action items
- Find out why the bug did not show deterministically
- Be more careful with sweeping Prow changes, with more end-to-end tests or a real staging Prow
Lessons
The behaviour was not deterministic, and the change was deployed on a Friday afternoon. Prow instances are set up differently, for example in how build clusters are defined, so another project's Prow running a version fine does not show it is safe for the Kubernetes Prow, which needs a staging environment that looks like it.
Sources
References
Related cases
IL-0356Every domain hosted on Railway returned HTTP 404 to new connections for five minutes because route lookups failedRailway · 2026same root cause · shared CI system
IL-0234An onboarding operator deployed without its scope file installed dev applications on other IONOS Kubernetes clustersIONOS Cloud · 2026same root cause · shared CI system
IL-0324Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database clusterWikimedia Foundation · 2025same root cause · shared CI system
IL-0552CircleCI's archival database ran out of storage, slowing workflow and job starts and the display of workflows in the UICircleCI · 2025same root cause · shared CI system
IL-0467Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hoursKubernetes project · 2019same root cause · shared CI system
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0466
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0466/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.