Incident Library
Applied filters ComponentAdmission webhookSymptomControl plane unresponsiveFactorMonitoring blind spotClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019
    Symptoms

    Jetstack was upgrading the master of a development cluster that many teams use to test their apps during the working day, a regional Google Kubernetes Engine (GKE) cluster in europe-west1. The upgrade ran through Jetstack's GKE Terraform pipeline and did not finish before the 20-minute Terraform timeout, while the GKE console still showed the cluster as upgrading.

    Upgrade stallControl plane unresponsiveNode recreation loopService outage
    Trigger

    A planned master upgrade of the shared development cluster. The same upgrade had been done on another pre-production environment earlier that day, and no API downtime was expected.

    Root cause
    Configuration change
    Component
    Admission webhookGKE control planeKubernetes API serverkubelet
    Contributing factor
    Excessive blast radiusNo health probesMonitoring blind spotAutomation amplification
    Detection

    The first sign was the master upgrade missing the 20-minute Terraform timeout; re-running the pipeline returned the kube-apiserver unhealthy error. The cause was identified with the help of Google Support.

    Remediation
    • Deleted the OPA ValidatingAdmissionWebhook resource during a window of API server access
    Action items
    • Limit the OPA webhook to the namespaces where the policy applies and development teams have access
    • Enable the webhook only for Ingress and Service resources, the only resources the policy validates
    • Add a liveness probe so OPA is restarted when it stops responding, and update the OPA documentation
    • Follow the open upstream Kubernetes issue about this behaviour
    Lessons

    With alerting on API server response times, the team might have seen CREATE and UPDATE requests slow down across the board as soon as the OPA-backed webhook was first deployed. Every workload needs probes: OPA looked so simple to deploy that the team skipped the Helm chart, which adds a livenessProbe for the admission controller containers among other adjustments to the tutorial configuration. Others had hit the same problem, and an upstream issue to improve it was still open.

    View case
Check your cluster for this

This case involves Admission webhook and GKE control plane. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
Admission webhookGKE control planeKubernetes API serverkubelet
Contributing factor
Automation amplificationExcessive blast radiusMonitoring blind spotNo health probes
Organisation
Jetstack
  1. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019