Incident Library
Applied filters SymptomControl plane lagFactorRetry stormOrganisationGoogle CloudClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022
    Symptoms

    From 03:03 US/Pacific on 14 June 2022, calls to the GKE API in us-central1 through gcloud and the Google Cloud Console were slow and returned errors. Retrying sometimes succeeded.

    High error rateHigh latencyControl plane lag
    Trigger

    An unexpected increase in backend traffic to the GKE API in us-central1.

    Root cause
    Capacity exhaustion
    Component
    GKE control plane
    Contributing factor
    Retry storm
    Remediation
    • scaled up the GKE control plane to handle the increased traffic
    Lessons

    Client retries against an overloaded management API add to the overload. When the cloud provider's control plane is degraded, running workloads may be fine while cluster operations fail, and the status page offered no workaround beyond advising customers to use another region if they could.

    View case
Check your cluster for this

This case involves GKE control plane. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
GKE control plane
Contributing factor
Retry storm
Organisation
Google Cloud
  1. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022