Cases14 cases

12 to 14 of 14

1. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node

   Jetstack2019

   Symptoms

   Jetstack was upgrading the master of a development cluster that many teams use to test their apps during the working day, a regional Google Kubernetes Engine (GKE) cluster in europe-west1. The upgrade ran through Jetstack's GKE Terraform pipeline and did not finish before the 20-minute Terraform timeout, while the GKE console still showed the cluster as upgrading.

   Upgrade stallControl plane unresponsiveNode recreation loopService outage

   Trigger

   A planned master upgrade of the shared development cluster. The same upgrade had been done on another pre-production environment earlier that day, and no API downtime was expected.

   Root cause

   Configuration change

   Component

   Admission webhookGKE control planeKubernetes API serverkubelet

   Contributing factor

   Excessive blast radiusNo health probesMonitoring blind spotAutomation amplification

   Detection

   The first sign was the master upgrade missing the 20-minute Terraform timeout; re-running the pipeline returned the kube-apiserver unhealthy error. The cause was identified with the help of Google Support.

   Remediation

   • Deleted the OPA ValidatingAdmissionWebhook resource during a window of API server access

   Action items

   • Limit the OPA webhook to the namespaces where the policy applies and development teams have access
   • Enable the webhook only for Ingress and Service resources, the only resources the policy validates
   • Add a liveness probe so OPA is restarted when it stops responding, and update the OPA documentation
   • Follow the open upstream Kubernetes issue about this behaviour

   Lessons

   With alerting on API server response times, the team might have seen CREATE and UPDATE requests slow down across the board as soon as the OPA-backed webhook was first deployed. Every workload needs probes: OPA looked so simple to deploy that the team skipped the Helm chart, which adds a livenessProbe for the admission controller containers among other adjustments to the tutorial configuration. Others had hit the same problem, and an upstream issue to improve it was still open.

   View case
2. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances

   Datadog2019

   Symptoms

   Users reported that their jobs were not starting, with some sign that image pulls were involved. Many pods across the cluster were in error states related to image pulls, while applications that were already running were not affected.

   Image pull failuresService outage

   Trigger

   A change to the permissions on a storage bucket that one application needed to read. The application could no longer start, and its DaemonSet, running on roughly a thousand nodes of the cluster, went into a crash loop with imagePullPolicy: Always, so every restart pulled the image again although it was already on the node.

   Root cause

   Capacity exhaustion

   Component

   Container registry and its credentialsNAT gatewayApplication databaseAdmission webhook

   Contributing factor

   Automation amplificationSingle point of failureUnknown limitHarmful manual actionExcessive blast radius

   Detection

   Users reported that their jobs were not starting; monitoring had not caught it first.

   Remediation

   • replaced the affected NAT instances so registry traffic came from new source IPs
   • reconfigured the static IPs and firewall rules so applications could reach the Cloud SQL database again

   Action items

   • Add an admission webhook that rejects floating image tags, starting with latest
   • Apply the image tag webhook to workload objects only, not to pods, after it blocked pods created by in-cluster controllers

   Lessons

   The speakers' takeaway is to be very careful with DaemonSets in large clusters: they say they had broken everything DaemonSets depend on, including DNS, Vault and the image repository, and that it is very easy to break things when managing them by the thousand. imagePullPolicy: Always made every crash-loop restart pull the image again. Sending all registry traffic through three NAT instances put the whole region behind three source IPs, which the registry rate-limited, and changing those IPs cut the connections to a critical database reached over the internet through them. An admission webhook on pods rejects the pods that controllers in the cluster create, where no user sees the error.

   View case
3. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down

   Universe2017

   Symptoms

   Shortly after a search reindex job was started, problems with the Kubernetes cluster hit Universe's reporting and search APIs and the Kubernetes cluster UI itself. The status page opened an investigation into reports of search and reporting being unavailable at 14:37 EDT on 2017-07-21.

   Service outageNode NotReadyNode recreation loop

   Trigger

   A change to the search index definition, which removed some fields from full-text indexing, was shipped, and a reindex job was started as a Kubernetes Job to apply it.

   Root cause

   Capacity exhaustion

   Component

   Jobs and CronJobsPod scheduling and placement

   Contributing factor

   Unverified assumptionAutomation amplificationExcessive blast radius

   Detection

   The team saw problems with the Kubernetes cluster shortly after starting the job, and the status page cites reports of search and reporting being unavailable.

   Remediation

   • removed the reindex Job from the scheduler
   • let the nodes auto-heal back to healthy

   Lessons

   restartPolicy: Never on the pod template does not stop a Kubernetes Job from retrying: a Job replaces failed pods and may run more than once, so a job that can exhaust a node will keep moving to new nodes.

   View case

By root cause6