IL-0008 Jetstack · 2019-09-05 (published)kubernetesgcp
Jetstack used the Open Policy Agent (OPA) project as an admission controller to enforce custom policies on multi-tenant clusters, through a ValidatingAdmissionWebhook. The webhook covered more namespaces and resources than the policy needed, including ConfigMaps in kube-system, and its OPA backend was not responding. OPA had no liveness probe, because it had been installed from the basic tutorial configuration rather than the Helm chart, which includes one.
With Google Support, the team reconstructed what happened. GKE finished upgrading the first master instance, which then received all API server traffic while the others were upgraded. During the upgrade of the second master, its API server had to run the PostStartHook for ca-registration, which updates the ConfigMap extension-apiserver-authentication in kube-system. That update went through the OPA webhook and timed out. The hook must complete for a master to pass its health check, so the second master went into a crash loop and the upgrade halted.
The resulting intermittent API downtime kept kubelets from reporting node health. GKE node auto-repair treats a node that reports no status for about 10 minutes as unhealthy and recreates it, so nodes were recreated again and again. Once the webhook was identified, the team used a window of API server access to delete the ValidatingAdmissionWebhook resource, and the cluster recovered. Afterwards the webhook was limited to the namespaces where the policy applies and development teams have access, and to Ingress and Service resources only, and OPA got a liveness probe. Disabling GKE node auto-repair was considered and rejected.
Jetstack was upgrading the master of a development cluster that many teams use to test their apps during the working day, a regional Google Kubernetes Engine (GKE) cluster in europe-west1. The upgrade ran through Jetstack's GKE Terraform pipeline and did not finish before the 20-minute Terraform timeout, while the GKE console still showed the cluster as upgrading.
Re-running the pipeline failed with this error: google_container_cluster.cluster: Error waiting for updating GKE master version: All cluster resources were brought up, but the cluster API is reporting that: component "kube-apiserver" from endpoint "gke-..." is unhealthy
During this time the API server timed out intermittently and teams could not deploy their applications. Then, while the investigation was going on, all nodes started being destroyed and recreated in an endless loop, and every tenant of the cluster lost service.
Symptom Upgrade stallControl plane unresponsiveNode recreation loop
Root cause Configuration change
The first sign was the master upgrade missing the 20-minute Terraform timeout; re-running the pipeline returned the kube-apiserver unhealthy error. The cause was identified with the help of Google Support.
With alerting on API server response times, the team might have seen CREATE and UPDATE requests slow down across the board as soon as the OPA-backed webhook was first deployed. Every workload needs probes: OPA looked so simple to deploy that the team skipped the Helm chart, which adds a livenessProbe for the admission controller containers among other adjustments to the tutorial configuration. Others had hit the same problem, and an upstream issue to improve it was still open.
IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time outDFDS · 2020same root cause · shared Kubernetes API server, Admission webhook
IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI servicesOpenAI · 2024same root cause · shared Kubernetes API server
IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clustersOpenAI · 2024same root cause · shared Kubernetes API server
IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1DigitalOcean · 2024same root cause · shared Kubernetes API server
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared kubelet
IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSetAqua Security · 2023same root cause · shared Kubernetes API server
IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second oneSpotify · 2019same root cause · shared GKE control plane
IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memoryNU.nl · 2019same root cause · shared kubelet