IL-0076 Fullstaq · 2019-06-01 (published)kubernetes
This is a meetup talk by Pieter Lange of Fullstaq, given as slides without notes. Dex ran with what the speaker calls bad defaults, an insane TTL, together with refreshing dashboards and an actual security bug. AuthRequest objects piled up in etcd until listing them timed out, and the speaker had seen hints of this earlier. The options he weighed were tuning etcd and API server timeouts, deleting the namespace and letting Kubernetes clean up, or deleting the objects directly in etcd.
He deleted the namespace. The reconciliation loops then listed all AuthRequests, timed out and started over, which drove master load to 60 and brought the masters down. He removed the kubernetes finalizer from the dex namespace by calling the API directly, with the kill-kube-ns tool, a step he marks NEVER DO THIS. That ended the outage, but about 0.5G of AuthRequests stayed in etcd, and he was fairly sure that creating a namespace named dex again would kill the cluster, so Dex was deployed to a new namespace with sane defaults. No business applications were affected.
Dex, the OIDC proxy that handled Kubernetes authentication, stored its data in the Kubernetes API (storage type kubernetes, inCluster: true) and had built up a huge number of AuthRequest objects. Listing all AuthRequests timed out, and without the list they could not be deleted.
After the dex namespace was deleted, the reconciliation loops kept listing all AuthRequests in order to delete them, timing out and repeating. Load on the masters rose to 60 and the masters went down. Monitoring could not be reached because authentication and port-forwarding no longer worked, while alerts kept arriving in Slack.
Root cause Capacity exhaustion
The speaker says he had got hints of the problem earlier. While the masters were down, plenty of alerts arrived in Slack, but monitoring could not be reached because authentication and port-forwarding no longer worked.
–
Dex with Kubernetes as its storage keeps every AuthRequest as an API object, and with a long TTL they pile up until the API cannot list them. Deleting a namespace that holds that many objects makes Kubernetes list them in a loop and can overload the masters. Monitoring that is reached through cluster authentication is lost when authentication is what broke.
IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API serverRender · 2024same root cause · shared Kubernetes API server, etcd
IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodesPalark · 2023same root cause · shared Kubernetes API server, etcd
IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the APICivis Analytics · 2019same root cause · shared Kubernetes API server, etcd
IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PARScaleway · 2024same root cause · shared Kubernetes API server
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Kubernetes API server
IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023same root cause · shared Kubernetes API server
IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limitZalando · 2019same root cause · shared Kubernetes API server
IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifestZalando · 2018same root cause · shared Kubernetes API server