Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases9 cases

8 to 9 of 9

  1. IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutes
    Google Cloud2022
    Symptoms

    From 10:10 US/Pacific on 2 June 2022, Vertex AI Online Prediction returned errors for 30% up to 100% of requests per region, depending on how each customer used the service.

    High error rateResources deletedService outage
    Trigger

    A faulty resource cleanup process marked Vertex Prediction Endpoints as deleted, globally.

    Root cause
    Software defect
    Component
    Model request routing
    Contributing factor
    Automation amplificationExcessive blast radius
    Detection

    The first public status update was posted at 13:58 US/Pacific and gave the start as 10:20 US/Pacific; the later mini incident report gives 10:10.

    Remediation
    • restored the Vertex Prediction Endpoints that had been marked as deleted
    View case
  2. IL-0143 GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS image
    Google Cloud2019
    Symptoms

    Some nodes in Google Kubernetes Engine clusters that had recently been upgraded showed an elevated number of kernel panics. The status page described it as kernel panics following an upgrade in GKE version.

    Node kernel panic
    Trigger

    Upgrading GKE node pools to one of the affected 1.13 or 1.14 versions.

    Root cause
    Software defect
    Component
    Node operating system imageLinux kernel
    Contributing factor
    –
    Remediation
    • advised affected users to downgrade to a patch release not on the affected list, with support helping Release Channel users
    • released GKE 1.13.11-gke.14, 1.13.12-gke.8, 1.14.7-gke.23 and 1.14.8-gke.12, which downgraded Container-Optimized OS to cos-73-11647-293-0
    • rolled the fixed versions out to node pools with auto-upgrade enabled; clusters on manual upgrades had to upgrade their nodes themselves
    Action items
    • Ship a GKE release with an upgraded kernel that fixes the panics
    Lessons

    Downgrading the node operating system image was used as a temporary mitigation until a kernel fix was available.

    View case
Check your cluster for this

These 9 cases involve Node operating system image and Calico. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause5

Configuration change1 case

Component
–
Contributing factor
–
Organisation
Google Cloud
  1. IL-0130 Deployments of large models from Vertex Model Garden failed with "Model server never became ready" in most regions for 13 days after a storage provisioning configuration change
    Google Cloud2024

Software defect5 cases

Component
Calico1Cloud load balancer1Credential broker for pods1Model inference server1kubelet1Linux kernel1Model request routing1Node operating system image1Pod termination1
Contributing factor
Automation amplification1Excessive blast radius1No headroom1Partial rollout1
Organisation
Google Cloud5
  1. IL-0131 A new kind of GenerateContent request made Vertex AI prediction API servers segfault, and once enough of them restarted at the same time the serving fleet could not carry the load
    Google Cloud2024
  2. IL-0141 gke-metadata-server versions 0.4.272 to 0.4.280 flooded Cloud Logging with "Unable to sync sandbox" debug logs on GKE clusters with Workload Identity, using up logging quota and adding billable usage
    Google Cloud2023
  3. IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico component
    Google Cloud2022
  4. IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutes
    Google Cloud2022
  5. IL-0143 GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS image
    Google Cloud2019

Dependency failure1 case

Component
GPU driverNode operating system imageNode pool
Contributing factor
Automation amplification
Organisation
Google Cloud
  1. IL-0021 GKE GPU nodes on Container-Optimized OS could not download NVIDIA drivers (403 Forbidden) and came up without working GPUs
    Google Cloud2024

Faulty deployment1 case

Component
Telemetry agent
Contributing factor
–
Organisation
Google Cloud
  1. IL-0142 GKE clusters on 1.24 and later wrote unexpected fluentbit error entries such as "Failed to get record: decoder: failed to decode payload: EOF" that used up the Logging API write quota for some customers
    Google Cloud2023

Capacity exhaustion1 case

Component
GKE control plane
Contributing factor
Retry storm
Organisation
Google Cloud
  1. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022