Cases1 case

1 to 1 of 1

1. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node

   Jetstack2019

   Symptoms

   Jetstack was upgrading the master of a development cluster that many teams use to test their apps during the working day, a regional Google Kubernetes Engine (GKE) cluster in europe-west1. The upgrade ran through Jetstack's GKE Terraform pipeline and did not finish before the 20-minute Terraform timeout, while the GKE console still showed the cluster as upgrading.

   Upgrade stallControl plane unresponsiveNode recreation loopService outage

   Trigger

   A planned master upgrade of the shared development cluster. The same upgrade had been done on another pre-production environment earlier that day, and no API downtime was expected.

   Root cause

   Configuration change

   Component

   Admission webhookGKE control planeKubernetes API serverkubelet

   Contributing factor

   Excessive blast radiusNo health probesMonitoring blind spotAutomation amplification

   Detection

   The first sign was the master upgrade missing the 20-minute Terraform timeout; re-running the pipeline returned the kube-apiserver unhealthy error. The cause was identified with the help of Google Support.

   Remediation

   • Deleted the OPA ValidatingAdmissionWebhook resource during a window of API server access

   Action items

   • Limit the OPA webhook to the namespaces where the policy applies and development teams have access
   • Enable the webhook only for Ingress and Service resources, the only resources the policy validates
   • Add a liveness probe so OPA is restarted when it stops responding, and update the OPA documentation
   • Follow the open upstream Kubernetes issue about this behaviour

   Lessons

   With alerting on API server response times, the team might have seen CREATE and UPDATE requests slow down across the board as soon as the OPA-backed webhook was first deployed. Every workload needs probes: OPA looked so simple to deploy that the team skipped the Helm chart, which adds a livenessProbe for the admission controller containers among other adjustments to the tutorial configuration. Others had hit the same problem, and an upstream issue to improve it was still open.

   View case

By root cause1