Incident Library
Applied filters ComponentRBAC and service accountsSymptomCredentials unavailableFactorSingle point of failureClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effect
    Webhook Relay2022
    Symptoms

    At 12:00 EST the health check system reported that the backend APIs, the frontend and the tunnels in the EU region were down. The team first took it for a transient error, such as an interrupted client request or a server evicted for memory or CPU pressure, but the main dashboard was offline as well.

    Service outageControl plane lagControl plane unresponsiveStale or inconsistent API readsCredentials unavailable
    Root cause
    Certificate or credential expiry
    Component
    GKE control planeKubernetes API serverNode poolRBAC and service accounts
    Contributing factor
    Limited debug accessSingle point of failure
    Detection

    The health check system sent a notification that the service was down; the team then found the main dashboard offline.

    Remediation
    • recreated the node pools by hand after the credential rotation failed to recreate them
    • duplicated the main cluster's services into a newly created GKE cluster
    • deleted the Kubernetes Service records in the old cluster so its load balancers could be attached to the new cluster
    • detached the StatefulSet from the old cluster and switched all writes to the main disk
    • exported the data from the temporary ClickHouse instance and inserted it into the main instance
    Action items
    • Change the deployment manifests so services can be switched between clusters quickly without a complicated persistent disk migration
    Lessons

    Recovery of existing infrastructure needs a time limit. Throughout the outage the old cluster looked about to recover, and the reluctance to give up on the salvage effort kept the team from starting over with a new cluster much sooner, which would have been the right call. On older GKE clusters, check the validity of the cluster CA certificate, for example by decoding masterAuth.clusterCaCertificate from gcloud container clusters describe and reading it with openssl x509. GKE's multiple availability zones made no difference in this failure.

    View case
Check your cluster for this

This case involves GKE control plane and Kubernetes API server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Certificate or credential expiry1 case

Component
GKE control planeKubernetes API serverNode poolRBAC and service accounts
Contributing factor
Limited debug accessSingle point of failure
Organisation
Webhook Relay
  1. IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effect
    Webhook Relay2022