Incident Library
Applied filters Root causeSoftware defectSymptomLog floodOrganisationGoogle CloudClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0141 gke-metadata-server versions 0.4.272 to 0.4.280 flooded Cloud Logging with "Unable to sync sandbox" debug logs on GKE clusters with Workload Identity, using up logging quota and adding billable usage
    Google Cloud2023
    Symptoms

    Google Kubernetes Engine (GKE) customers with Workload Identity enabled saw a high logging rate. Debug log lines containing the string "Unable to sync sandbox" were ingested into Cloud Logging, used up the Cloud Logging ingestion quota and, beyond the free monthly allotment, produced extra billable usage. Workload Identity is opt-in on GKE Standard and always enabled on GKE Autopilot.

    Log floodHigh latency
    Trigger

    GKE clusters ran gke-metadata-server versions 0.4.272 to 0.4.280, which contained an incorrect configuration that wrote "Unable to sync sandbox" debug logs at a high rate.

    Root cause
    Software defect
    Component
    Credential broker for pods
    Contributing factor
    –
    Detection

    The first status update was posted at 13:32 US/Pacific on 2023-11-02, and the updates told customers how to check a cluster from the version annotation on the gke-metadata-server DaemonSet.

    Remediation
    • rolled out a fix across the GKE fleet so the excess logs were no longer ingested into Cloud Logging
    • offered GKE Rapid channel customers a control plane upgrade to 1.28.2-gke.1157000 or above, or 1.27.7-gke.1038000 or above, as a workaround
    View case
Check your cluster for this

This case involves Credential broker for pods. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
Credential broker for pods
Contributing factor
–
Organisation
Google Cloud
  1. IL-0141 gke-metadata-server versions 0.4.272 to 0.4.280 flooded Cloud Logging with "Unable to sync sandbox" debug logs on GKE clusters with Workload Identity, using up logging quota and adding billable usage
    Google Cloud2023