Incident Library

IL-0401 Kubernetes project · 2019-05-30kubernetesgcp

Prow jobs on the k8s-prow-build GKE cluster failed and merges stalled as nodes hit IOPS throttling and kernel panics

Prow runs Kubernetes project test jobs in a 100-node GKE cluster, k8s-prow-build, whose nodes had eight CPUs, 52 GB of RAM and 250 GB disks. On GCP, IOPS are limited in proportion to disk size, which gave 187.5 IOPS per instance. Most nodes barely touched the disk, but nodes running some workloads spiked to large multiples of that, and test load had grown over time. Ahead of the 1.15 code freeze a spike in usage hit hard throttling, and I/O became so slow that pods failed, kubelets crashed and nodes had kernel panics. Only a minority of nodes were affected at any time, but a job needs many pods to complete before a PR can merge, so merging nearly stopped for at least seven hours and the code freeze slipped a day.

The problem was reported on Slack at 13:26. At 14:11 the team found many problems with the nodes and test pods, and at 14:43 an engineer brought in from the node team identified severe I/O throttling as the root cause. The oncaller was told how to add capacity and got additional disk quota by 15:14, but capacity was not increased until 17:10, when 55 nodes with 500 GB disks were brought up, within the quota headroom. From 17:41 the remaining capacity was migrated gradually from the 250 GB node pool to the 500 GB pool by draining old nodes; the migration finished at 23:55 and Prow was merging at its usual rate again by 07:00. A global `max_concurrency` for plank was suggested as another way to throttle job load but was held back while the new nodes took effect. Because of an unrelated failure, most prow/tide monitoring for the week before the incident is lost, so when the cluster first became faulty is not known.

Severity
–
Scope
Prow CI jobs failed, were aborted or did not upload artifacts; at peak 60 PRs were stuck in the merge queue and no PRs merged for many hours; the Kubernetes 1.15 code freeze slipped a day
Users affected
Kubernetes contributors whose PRs depended on Prow
Region
us-central1-f
Time to detect
–
Time to resolve
–

Symptoms

On 30 May 2019, in the run-up to the Kubernetes 1.15 code freeze, Prow test jobs in the k8s-prow-build cluster failed in surprising ways or were aborted partway through. Jobs were reported to GitHub as failed while Deck still showed them as running, because pods exited without uploading `finished.json`, the file most Prow tooling uses to tell a job's state.

One terminated job pod showed `State: Terminated`, `Exit Code: 0`, and `Started` and `Finished` both as `Mon, 01 Jan 0001 00:00:00 +0000`. Artifacts were often not uploaded, so Spyglass and Testgrid could not see that jobs had finished, and spurious Testgrid flake alerts fired. Pods and nodes were in odd states: pods were being evicted by something the team did not control, pods were not being killed, something had tried to kill one job pod 2,420 times over four days, pods sat in `PodInitializing` for half an hour, and kubelets were in reboot loops.

Nodes in the containerd pool went `NotReady` (for example `gke-prow-containerd-pool-99179761-02q6 NotReady <none> 14d v1.13.5-gke.10`), and the set of NotReady nodes kept changing. The team first suspected kubelet to CRI RPC problems and a deadlock when deleting pods, and then found kernel panics. The disks' estimated limits were 187.50 read and 375.00 write sustained random IOPS and 30 MB/s throughput, while throttled nodes were reaching 80MB/s and more than 450 IOPS; the throttled disk operations graph showed the problem had grown much worse over the previous day or two.

Merging slowed to a stop. At peak 60 PRs were stuck in the merge queue, the last merge into kubernetes/kubernetes was 5 hours old by 16:05, and no PRs merged for many hours. Tide itself was working, but CI was so red that the queue drained slowly. In the GKE chat the problem was described as long-running and previously written off as COS / containerd issues.

Symptom Node NotReadyNode kernel panicPods stuck in TerminatingThroughput degradation

Root cause

Root cause Capacity exhaustion

Component Cloud block storageNode root diskkubeletCI system
Contributing factor Monitoring blind spotIgnored early signal
Trigger
A spike in test load before the Kubernetes 1.15 code freeze pushed disk I/O on the build nodes far beyond the IOPS that GCP allows for 250 GB disks.

Detection

Several users reported jobs that GitHub showed as failed but Deck showed as still running, and the first report came on Slack at 13:26. Alerting for these failures did not exist or did not work.

Remediation

Action items

Lessons

Prow metrics were not recorded before the incident, which made it hard to understand, and alerting for these failures did not exist or did not work. After the cause was found and a fix devised, nothing happened for another two hours. The team counts it as luck that someone who recognised IOPS as the problem happened to be around to look.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cloud block storage, Node root disk, kubelet among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.