IL-0021 Google Cloud · 2024-03-12kubernetesgcpai-infra
GKE nodes on Container-Optimized OS download the NVIDIA GPU driver from a storage bucket owned by the partner that supplies the drivers. On 12 March 2024 access to that bucket failed. Every affected GPU node that was created or recreated got a 403 Forbidden on the driver download and ended up without a working driver, for 6 hours 55 minutes, from 08:00 to 14:55 US/Pacific. The root cause in the final mini incident report comes from a preliminary analysis.
Automatic node recreation attempts the GPU driver download again, so to limit the impact Google halted automatic node recreations until the issue was mitigated, and worked with the partner to resolve the underlying access problem. Other GKE features kept working normally. After the fix, customers who had disabled auto-repair might need to recreate or restart the affected nodes themselves to get working GPUs back.
From 08:00 US/Pacific on 12 March 2024, GKE GPU nodes running Container-Optimized OS (COS) that were newly created or recreated came up with GPU drivers that did not work, so workloads that used the GPUs could not run. GKE showed elevated errors.
On the GPU node, installing the driver failed with the message "Failed to download GPU driver installer, status: 403 Forbidden". In some cases the failed download left the node unavailable.
Only some combinations were hit: T4, L4, H100 80GB and A100 GPUs on COS milestone 105 or later, installing GPU driver versions R525 and above, in all GPU regions and zones. Driver downloads for P4, P100, V100 and K80 GPUs kept working. Google had no workaround other than to avoid recreating existing GPU nodes.
Symptom GPU unavailableNode NotReady
Root cause Dependency failure
The report does not say how Google detected the issue. Affected users saw "Failed to download GPU driver installer, status: 403 Forbidden" on the GPU node when installing the driver.
–
The status page report states no lessons or follow-up actions.
IL-0039 New Kubernetes worker nodes could not join any cluster because a package URL hardcoded in kops nodeup brokeFREE NOW · 2019same root cause · shared Node operating system image, Node pool
IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiodTetrate (unnamed U.S. federal customer) · 2025 · sev1same root cause · shared Node operating system image
IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training clusterImbue · 2024same root cause · shared GPU driver
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.