Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0021 GKE GPU nodes on Container-Optimized OS could not download NVIDIA drivers (403 Forbidden) and came up without working GPUs
    Google Cloud2024
    Symptoms

    From 08:00 US/Pacific on 12 March 2024, GKE GPU nodes running Container-Optimized OS (COS) that were newly created or recreated came up with GPU drivers that did not work, so workloads that used the GPUs could not run. GKE showed elevated errors.

    GPU unavailableNode NotReady
    Root cause
    Dependency failure
    Component
    GPU driverNode operating system imageNode pool
    Contributing factor
    Automation amplification
    Detection

    The report does not say how Google detected the issue. Affected users saw "Failed to download GPU driver installer, status: 403 Forbidden" on the GPU node when installing the driver.

    Remediation
    • Halted automatic node recreations, which attempt GPU driver downloads, until the issue was mitigated
    • Worked with the driver partner to resolve access to the driver download bucket
    Lessons

    The status page report states no lessons or follow-up actions.

    View case
Check your cluster for this

This case involves GPU driver and Node operating system image. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Dependency failure1 case

Component
GPU driverNode operating system imageNode pool
Contributing factor
Automation amplification
Organisation
Google Cloud
  1. IL-0021 GKE GPU nodes on Container-Optimized OS could not download NVIDIA drivers (403 Forbidden) and came up without working GPUs
    Google Cloud2024