Cases11 cases

4 to 8 of 11

1. IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in production

   Sunwoo Park2023

   Symptoms

   On 2023-05-18 a Slack alert arrived at 20:01 and Istio stopped working properly in the production GKE cluster. The istiod error logs said that the v1beta1.Ingress and v1beta1.ValidatingWebhookConfiguration resources could not be found.

   Service outage

   Trigger

   GKE automatically upgraded the production cluster's control plane from 1.22.17-gke.7500 to 1.23.17-gke.1700 at 19:52 on 2023-05-18.

   Root cause

   Breaking change in an upgrade

   Component

   IstioGKE control planeAdmission webhook

   Contributing factor

   Deferred upgrades

   Detection

   A Slack alert at 20:01 on 2023-05-18, nine minutes after the control plane upgrade. The istiod error logs showed that the v1beta1 Ingress and ValidatingWebhookConfiguration resources could not be found.

   Remediation

   • upgraded Istio from 1.7.4 to 1.13.3

   Lessons

   The first diagnosis, end of support for Istio, was wrong; the real cause was found by matching the alert time against the cluster's UPGRADE_MASTER operations in Logs Explorer. The author recommends checking the version compatibility matrices of Kubernetes, Istio and other packages, keeping a production version for at least six months and upgrading three months before end of support, keeping the test environment on the latest version, backing up before upgrades, managing deployments and versions with a tool such as Argo CD, and testing upgrades in a test environment before production.

   View case
2. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse

   Google Cloud2022

   Symptoms

   From 03:03 US/Pacific on 14 June 2022, calls to the GKE API in us-central1 through gcloud and the Google Cloud Console were slow and returned errors. Retrying sometimes succeeded.

   High error rateHigh latencyControl plane lag

   Trigger

   An unexpected increase in backend traffic to the GKE API in us-central1.

   Root cause

   Capacity exhaustion

   Component

   GKE control plane

   Contributing factor

   Retry storm

   Remediation

   • scaled up the GKE control plane to handle the increased traffic

   Lessons

   Client retries against an overloaded management API add to the overload. When the cloud provider's control plane is degraded, running workloads may be fine while cluster operations fail, and the status page offered no workaround beyond advising customers to use another region if they could.

   View case
3. IL-0403 mybinder.org could not create pods for hours as the GKE gke-resource-quotas quota wrongly reported 15k pods in use

   Binder Project2022

   Symptoms

   On 2 June 2022 from 10:00 CET, mybinder.org stopped launching any new pods, and nobody on the team noticed until a user reported a Binder outage in the Matrix channel at 21:00.

   Deploys failingService outage

   Root cause

   Software defect

   Component

   GKE control plane

   Contributing factor

   Monitoring blind spot

   Detection

   A user reported a Binder outage in the Matrix channel at 21:00, and a team member who saw it found that pods had not been launching for several hours.

   Remediation

   • Deleted the gke-resource-quotas ResourceQuota, after which pods could be created again

   Action items

   • Address uptime monitoring and alerting

   Lessons

   It took 11 hours to realise there was a major outage. Once the right object was known, deleting it fixed the problem quickly, and the team was lucky that someone with the skills and permissions to make the change was at their computer at 11pm their time.

   View case
4. IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effect

   Webhook Relay2022

   Symptoms

   At 12:00 EST the health check system reported that the backend APIs, the frontend and the tunnels in the EU region were down. The team first took it for a transient error, such as an interrupted client request or a server evicted for memory or CPU pressure, but the main dashboard was offline as well.

   Service outageControl plane lagControl plane unresponsiveStale or inconsistent API readsCredentials unavailable

   Root cause

   Certificate or credential expiry

   Component

   GKE control planeKubernetes API serverNode poolRBAC and service accounts

   Contributing factor

   Limited debug accessSingle point of failure

   Detection

   The health check system sent a notification that the service was down; the team then found the main dashboard offline.

   Remediation

   • recreated the node pools by hand after the credential rotation failed to recreate them
   • duplicated the main cluster's services into a newly created GKE cluster
   • deleted the Kubernetes Service records in the old cluster so its load balancers could be attached to the new cluster
   • detached the StatefulSet from the old cluster and switched all writes to the main disk
   • exported the data from the temporary ClickHouse instance and inserted it into the main instance

   Action items

   • Change the deployment manifests so services can be switched between clusters quickly without a complicated persistent disk migration

   Lessons

   Recovery of existing infrastructure needs a time limit. Throughout the outage the old cluster looked about to recover, and the reluctance to give up on the salvage effort kept the team from starting over with a new cluster much sooner, which would have been the right call. On older GKE clusters, check the validity of the cluster CA certificate, for example by decoding masterAuth.clusterCaCertificate from gcloud container clusters describe and reading it with openssl x509. GKE's multiple availability zones made no difference in this failure.

   View case
5. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node

   Jetstack2019

   Symptoms

   Jetstack was upgrading the master of a development cluster that many teams use to test their apps during the working day, a regional Google Kubernetes Engine (GKE) cluster in europe-west1. The upgrade ran through Jetstack's GKE Terraform pipeline and did not finish before the 20-minute Terraform timeout, while the GKE console still showed the cluster as upgrading.

   Upgrade stallControl plane unresponsiveNode recreation loopService outage

   Trigger

   A planned master upgrade of the shared development cluster. The same upgrade had been done on another pre-production environment earlier that day, and no API downtime was expected.

   Root cause

   Configuration change

   Component

   Admission webhookGKE control planeKubernetes API serverkubelet

   Contributing factor

   Excessive blast radiusNo health probesMonitoring blind spotAutomation amplification

   Detection

   The first sign was the master upgrade missing the 20-minute Terraform timeout; re-running the pipeline returned the kube-apiserver unhealthy error. The cause was identified with the help of Google Support.

   Remediation

   • Deleted the OPA ValidatingAdmissionWebhook resource during a window of API server access

   Action items

   • Limit the OPA webhook to the namespaces where the policy applies and development teams have access
   • Enable the webhook only for Ingress and Service resources, the only resources the policy validates
   • Add a liveness probe so OPA is restarted when it stops responding, and update the OPA documentation
   • Follow the open upstream Kubernetes issue about this behaviour

   Lessons

   With alerting on API server response times, the team might have seen CREATE and UPDATE requests slow down across the board as soon as the OPA-backed webhook was first deployed. Every workload needs probes: OPA looked so simple to deploy that the team skipped the Helm chart, which adds a livenessProbe for the admission controller containers among other adjustments to the tutorial configuration. Others had hit the same problem, and an upstream issue to improve it was still open.

   View case

By root cause6