IL-0505 Binder Project · 2019-02-20kubernetesgcp
kubectl logs failed on GKE with "No SSH tunnels currently open" and mybinder.org build logs stopped reaching users
On 20 February 2019 one node of the mybinder.org GKE cluster, gke-prod-a-users-b921bb88-rd5d, had lost its ExternalIP and InternalIP in its node status, and the Kubernetes API server repeatedly failed to get addresses for it. From 08:03 CET kubectl logs, which reached the nodes' port 10250 through SSH tunnels from the master, failed for all pods with "No SSH tunnels currently open", so build logs no longer reached users and a deploy failed. The write-up says the cause appears to be a bug in the Kubernetes master and/or the GKE service itself, which made the root cause very hard to identify; upgrading the master and bringing in new nodes did not fix it.
After the API server log pointed at the node, it was cordoned so it would be removed from the pool once empty. At 13:30, with the node drained and removed, kubectl logs worked again. Throughout the incident and the cluster upgrade, launch success never fell below 97.5% and was only very briefly below 100%.
- Severity
- –
- Scope
- mybinder.org: users saw no build logs for new builds and kubectl logs failed for all pods; builds and launches kept working, with launch success never below 97.5%
- Users affected
- mybinder.org users running new builds
- Region
- –
- Time to detect
- –
- Time to resolve
- 5h 27m
Symptoms
On 20 February 2019 at 08:03 CET, `kubectl logs -f <pod>` stopped working for every pod in the mybinder.org cluster on GKE. Because pod logs could not be read, build logs were no longer forwarded to users, although the builds themselves ran normally.
The command failed with "Error from server: Get https://10.128.0.6:10250/containerLogs/prod/jupyter-.../notebook?follow=true: No SSH tunnels currently open. Were the targets able to accept an ssh-key for user "gke-df315ed616f010764e03"?". At 10:12 a deploy of a minor version bump failed because the SSH tunnels could not be created. The same failure had happened once before, on 2019-01-08, lasted 24 hours and went away without ever being diagnosed.
A planned cluster upgrade began at 10:20 in the hope that a new master and new nodes would fix it, but at 11:17, with the new nodes running and the old ones still draining, kubectl logs failed with the same error. Following the GKE troubleshooting documentation, the team checked the master's SSH access to the nodes: the ssh-key metadata was not too full, the key for gke-df315ed616f010764e03 was in authorized_keys on all nodes, firewall rules allowed SSH from the master, and journalctl showed no SSH attempt from the master at all. Nothing on the nodes was stopping the master from opening a tunnel.
At 12:00 the API server log, fetched through `kubectl proxy` from /logs/kube-apiserver.log, showed a repeated "Failed to getAddresses: no preferred addresses found; known addresses: [{Hostname gke-prod-a-users-b921bb88-rd5d}]" among many repeats of the SSH tunnel error. Listing node addresses with `kubectl get node -o json | jq '.items[] | [.status.addresses[].address]'` showed that node gke-prod-a-users-b921bb88-rd5d had no InternalIP or ExternalIP, only an empty string and its hostname.
Symptom Monitoring outageDeploys failing
Root cause
Root cause Cloud provider outage
- Trigger
- –
Detection
–
Remediation
- Cordoned node gke-prod-a-users-b921bb88-rd5d so it would be removed from the pool once empty
- Drained the node and removed it from the pool, after which kubectl logs worked again
Action items
- Test that kubectl logs works and notify when it does not
- Add the command for retrieving API server logs to the SRE guide
- Describe in the SRE guide how to identify suspicious node networking status
Lessons
The root cause was very difficult to identify, partly because it appears to sit in the Kubernetes master or the GKE service itself. Capturing `kubectl describe node` for the suspicious node would have allowed a comparison with healthy nodes, and a node that reports no ExternalIP or InternalIP in its status was likely a direct symptom of the problem and can be used as a diagnostic in the future.
Sources
References
Related cases
IL-0463Secure Connect connectors timed out after a GKE control plane upgrade left Harness's internal load balancer unreachableHarness · 2026same root cause · shared GKE control plane
IL-0542Twingate authentication and authorization requests failed at higher rates after traffic moved onto a smaller GKE clusterTwingate · 2025same root cause · shared GKE control plane
IL-0084Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPUNamespace · 2024same root cause · shared Kubernetes API server
IL-0014A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPAloveholidays · 2019same root cause · shared GKE control plane
IL-0469The managed Azure Kubernetes API server for the data100 hub stopped answering, so the stopped hub pod was not recreatedUC Berkeley · 2019same root cause · shared Kubernetes API server
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0505
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0505/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.