Incident Library
Applied filters FactorAutomation amplificationOrganisationGoogle CloudClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0021 GKE GPU nodes on Container-Optimized OS could not download NVIDIA drivers (403 Forbidden) and came up without working GPUs
    Google Cloud2024
    Symptoms

    From 08:00 US/Pacific on 12 March 2024, GKE GPU nodes running Container-Optimized OS (COS) that were newly created or recreated came up with GPU drivers that did not work, so workloads that used the GPUs could not run. GKE showed elevated errors.

    GPU unavailableNode NotReady
    Root cause
    Dependency failure
    Component
    GPU driverNode operating system imageNode pool
    Contributing factor
    Automation amplification
    Detection

    The report does not say how Google detected the issue. Affected users saw "Failed to download GPU driver installer, status: 403 Forbidden" on the GPU node when installing the driver.

    Remediation
    • Halted automatic node recreations, which attempt GPU driver downloads, until the issue was mitigated
    • Worked with the driver partner to resolve access to the driver download bucket
    Lessons

    The status page report states no lessons or follow-up actions.

    View case
  2. IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutes
    Google Cloud2022
    Symptoms

    From 10:10 US/Pacific on 2 June 2022, Vertex AI Online Prediction returned errors for 30% up to 100% of requests per region, depending on how each customer used the service.

    High error rateResources deletedService outage
    Trigger

    A faulty resource cleanup process marked Vertex Prediction Endpoints as deleted, globally.

    Root cause
    Software defect
    Component
    Model request routing
    Contributing factor
    Automation amplificationExcessive blast radius
    Detection

    The first public status update was posted at 13:58 US/Pacific and gave the start as 10:20 US/Pacific; the later mini incident report gives 10:10.

    Remediation
    • restored the Vertex Prediction Endpoints that had been marked as deleted
    View case
Check your cluster for this

These 2 cases involve GPU driver and Model request routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Dependency failure1 case

Component
GPU driverNode operating system imageNode pool
Contributing factor
Automation amplification
Organisation
Google Cloud
  1. IL-0021 GKE GPU nodes on Container-Optimized OS could not download NVIDIA drivers (403 Forbidden) and came up without working GPUs
    Google Cloud2024

Software defect1 case

Component
Model request routing
Contributing factor
Automation amplificationExcessive blast radius
Organisation
Google Cloud
  1. IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutes
    Google Cloud2022