Incident Library

Organisation

Google Cloud

Google Cloud · google-cloud

9 cases · 2019 to 2024

Root cause Software defect5Capacity exhaustion1Configuration change1Dependency failure1Faulty deployment1
Component Node operating system image2Calico1Cloud load balancer1Credential broker for pods1GKE control plane1GPU driver1Model inference server1kubelet1Linux kernel1Model request routing1Node pool1Pod termination1Telemetry agent1
Contributing factor Automation amplification2Excessive blast radius1No headroom1Partial rollout1Retry storm1
Symptom High error rate3High latency3Log flood2Control plane lag1Deploys failing1GPU unavailable1Intermittent timeouts1Node kernel panic1Node NotReady1Pods Pending1Pods stuck in Terminating1Resources deleted1Service outage1

Cases

  1. IL-0130 Deployments of large models from Vertex Model Garden failed with "Model server never became ready" in most regions for 13 days after a storage provisioning configuration changeGoogle Cloud · 2024

    Root cause Configuration change

  2. IL-0131 A new kind of GenerateContent request made Vertex AI prediction API servers segfault, and once enough of them restarted at the same time the serving fleet could not carry the loadGoogle Cloud · 2024

    Root cause Software defect

  3. IL-0021 GKE GPU nodes on Container-Optimized OS could not download NVIDIA drivers (403 Forbidden) and came up without working GPUsGoogle Cloud · 2024

    Root cause Dependency failure

  4. IL-0141 gke-metadata-server versions 0.4.272 to 0.4.280 flooded Cloud Logging with "Unable to sync sandbox" debug logs on GKE clusters with Workload Identity, using up logging quota and adding billable usageGoogle Cloud · 2023

    Root cause Software defect

  5. IL-0142 GKE clusters on 1.24 and later wrote unexpected fluentbit error entries such as "Failed to get record: decoder: failed to decode payload: EOF" that used up the Logging API write quota for some customersGoogle Cloud · 2023

    Root cause Faulty deployment

  6. IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico componentGoogle Cloud · 2022

    Root cause Software defect

  7. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worseGoogle Cloud · 2022

    Root cause Capacity exhaustion

  8. IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutesGoogle Cloud · 2022

    Root cause Software defect

  9. IL-0143 GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS imageGoogle Cloud · 2019

    Root cause Software defect