Incident Library
Applied filters ComponentGKE control planeSymptomHigh error rateDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0542 Twingate authentication and authorization requests failed at higher rates after traffic moved onto a smaller GKE cluster
    Twingate2025
    Symptoms

    On 21 March 2025, between 21:10 and 21:41 UTC, part of the authentication and authorization requests to Twingate services failed with elevated error rates.

    High error rate
    Trigger

    Twingate saw anomalies in one of its GKE clusters and scaled down the deployments in that cluster to protect the rest of the system, which moved its traffic to the other clusters.

    Root cause
    Cloud provider outage
    Component
    Cloud load balancerGKE control planeHorizontal pod autoscalerCluster autoscaler
    Contributing factor
    Retry stormNo headroomOverlapping failuresHarmful manual action
    Detection

    Twingate observed anomalies in one of its GKE clusters at the start of the incident.

    Remediation
    • Reintroduced capacity in the cluster that had been scaled down, so traffic was rebalanced across regions
    Action items
    • Keep working with the cloud provider to find the cause of the unexpected timeouts and of the cluster update's impact
    • Run deployments in all regions with the same number of replicas in the HPA configuration
    • Raise the maximum node counts of the cluster node autoscalers to leave more room to scale up
    • Configure the HPA to scale up faster
    View case
  2. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022
    Symptoms

    From 03:03 US/Pacific on 14 June 2022, calls to the GKE API in us-central1 through gcloud and the Google Cloud Console were slow and returned errors. Retrying sometimes succeeded.

    High error rateHigh latencyControl plane lag
    Trigger

    An unexpected increase in backend traffic to the GKE API in us-central1.

    Root cause
    Capacity exhaustion
    Component
    GKE control plane
    Contributing factor
    Retry storm
    Remediation
    • scaled up the GKE control plane to handle the increased traffic
    Lessons

    Client retries against an overloaded management API add to the overload. When the cloud provider's control plane is degraded, running workloads may be fine while cluster operations fail, and the status page offered no workaround beyond advising customers to use another region if they could.

    View case
Check your cluster for this

These 2 cases involve GKE control plane and Cloud load balancer. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Cloud provider outage1 case

Component
Cloud load balancerCluster autoscalerGKE control planeHorizontal pod autoscaler
Contributing factor
Harmful manual actionNo headroomOverlapping failuresRetry storm
Organisation
Twingate
  1. IL-0542 Twingate authentication and authorization requests failed at higher rates after traffic moved onto a smaller GKE cluster
    Twingate2025

Capacity exhaustion1 case

Component
GKE control plane
Contributing factor
Retry storm
Organisation
Google Cloud
  1. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022