Incident Library
Applied filters ComponentNode poolSymptomHigh latencyFactorMonitoring blind spotClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1
    GitLab2021
    Symptoms

    At 08:05 UTC on 2021-06-21 the API service's Apdex (the share of requests that finish within a target time) and Rails request queuing began trending the wrong way. At 09:01 the team found the API deployment unhealthy: it needed more pods for the day's growing traffic, but no new nodes could be added to give it pod capacity. Users saw slow responses; the review found no sign that error rates rose.

    Scale-up failureHigh latencyPods PendingDeploys failing
    Trigger

    Google Cloud ran out of SSD persistent disk stock in us-east1, the region GitLab.com runs in, while the API service needed more nodes for rising user traffic.

    Root cause
    Cloud provider outage
    Component
    Cluster autoscalerNode pool
    Contributing factor
    Monitoring blind spotLimited debug accessOverlapping failures
    Detection

    There was no alert on resource availability. Apdex and Rails queuing began trending the wrong way at 08:05 UTC and an engineer declared the incident in Slack at 08:43; finding the cause took a lot of log digging: cluster autoscaler errors about node pools in backoff, then ZONE_RESOURCE_POOL_EXHAUSTED at the instance-group level, and finally Google Cloud support naming SSD as the exhausted resource.

    Remediation
    • added a node pool that uses pd-standard disks
    • scaled the new node pool by hand across all three zones to relieve the API
    • brought more pd-standard node pools online for the other blocked workloads
    • stopped production deployments and opened a Google Cloud support case about the cluster autoscaler not working with the new node pools
    • changed node pools and nodeSelectors so the remaining workloads ran on pd-standard nodes
    • removed the SSD-backed node pools after a day without problems
    Action items
    • Move Kubernetes services to pd-standard disks (delivery#1837)
    • Capture resource exhaustion errors as a metric
    Lessons

    A cloud region can run out of one specific resource, here SSD persistent disks, and the only error may be a generic ZONE_RESOURCE_POOL_EXHAUSTED that does not name it. GitLab could neither monitor nor reserve resource availability, so detection took 22 hours and the cause came from Google Cloud support. The workloads ran fine on pd-standard disks. A second problem found on the way, a cluster autoscaler that does not fall back to another matching node pool when one cannot scale, slowed the move to the new node pools.

    View case
Check your cluster for this

This case involves Cluster autoscaler and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Cloud provider outage1 case

Component
Cluster autoscalerNode pool
Contributing factor
Limited debug accessMonitoring blind spotOverlapping failures
Organisation
GitLab
  1. IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1
    GitLab2021